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Introduction 


MultiScheme  is  a  fully  operational  parallel-programming  system  based  upon 
the  Scheme  dialect  of  Lisp.  Like  its  Lisp  ancestors,  MultiScheme  provides 
a  conducive  environment  for  prototyping  and  testing  new  linguistic  struc¬ 
tures  and  programming  methodologies.  MultiScheme  supports  a  diverse 
community  of  users  who  have  a  wide  range  of  interests  in  parallel  pro¬ 
gramming.  MultiScheme’s  flexible  support  for  system-based  experiments 
in  parallel  processing  has  enabled  it  to  serve  as  a  development  vehicle  for 
university  and  industrial  research.  At  the  same  time,  MultiScheme  is  suffi¬ 
ciently  robust,  and  supports  a  sufficiently  wide  range  of  parallel-processing 
applications,  that  it  has  become  the  base  for  a  commercial  product,  the 
Butterfly  Lisp  System  produced  by  BBN  Advanced  Computers,  Inc. 

This  report  pursues  three  interrelated  topics  that  are  elucidated  by  the 
MultiScheme  work.  First,  we  describe  extensions  that  suffice  to  transform 
sequential  Scheme  into  a  powerful  vehicle  for  multiprocessing.  Gratifyingly, 
only  a  few  extensions  are  needed.  The  two  most  important  extensions  are 
the  inclusion  of  placeholders  —  data  types  used  for  representing  values  that 
have  not  yet  been  computed  —  and  the  use  of  the  garbage  collector  as  a 
visible  part  of  the  language.  Placeholders  are  the  implementation  ba.se  for 
the  future  construct  of  Halstead’s  Multilisp[26j  language.  In  MultiScheme, 
however,  they  are  strictly  a  data  type  rather  than  a  combination  of  data 
type,  task,  and  syntax  as  in  Multilisp.  Most  of  the  other  extensions  have 
tjeen  familiar  as  “folklore”  within  the  Lisp  community.  Our  essential  contri¬ 
bution  here  is  to  present  carefully-documented,  robust  implementations  and 
to  demonstrate  the  synergy  of  including  these  within  one  coherent  system. 
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In  particular,  the  combination  of  placeholders,  controlled  garba^'e  collec¬ 
tion,  and  a  few  operations  for  handling  processor  synchronization  forms  a 
sufficient  base  for  implementing  the  MultiScheme  scheduler. 

Secondly,  we  argue  that  even  in  the  sequential  context,  adding  these  con 
structs  significantly  ('xtends  the  expressive  power  of  the  Scheme  language. 
Piaceiadders,  for  example,  allow  one  to  embed  logic  variables  within  sfaji 
(lard  Scheme  data  structures.  They  also  provide  a  mechanism  for  mingling 
iiormal-oi(ler  evaluation  with  an  otherwise  applicative-ordt'r  interpreter. 
The  \'i.'ible  aspects  of  the  garbtige  colh.'ctor  ]>rovi(le  a  con\'enieiit  Ijase  for 
the  iiiiplenient  at  ion  of  popul;itions[-4S]  (also  known  as  "wt'i'ik  sets  ').  associ¬ 
ation  table>  (an  extension  of  Lisp's  "'property  lists"  to  f)bjecrs  other  than 
symbols),  and  producer-to-consumer  links  (vital  to  automatic  deallocation 
of  tasks ). 

Finally,  we  demonstrate  how  MultiScheme  allows  one  to  attack  a  wide 
range  of  problem  domains  and  to  investigate  many  different  methodologies 
for  parallel  prograntming.  For  example,  by  measuring  the  time  required  to 
compute  the  future  state  of  an  8-body  gravitational  system  we  demonstrate 
how  appropriately  adding  futures  can  convert  a  serial  solution  to  a  parallel 
solution  that  runs  G.G  times  faster  (on  a  16  processor  machine).  By  study¬ 
ing  the  utilization  of  the  processors  in  this  solution  we  arc  led  to  a  varitint 
on  tilt'  dynamic  dataflow  methodology  for  solving  the  same  pnffilem.  By 
expressing  the  same  problem  in  this  new  methodology  we  incur  17*/I  over¬ 
head  in  sinudating  the  dataflow  processor,  but  are  able  solve  the  iiroblem 
G.9  times  faster  than  the  original  serial  solution.  Other  mt'thodologies  that 
can  be  conveniently  expressed  within  MultiScheme  include  fork/join  struc¬ 
tures.  Gabriel  and  McCarthy’s  QLambda[23j,  and  the  Butterfly  Unifora 
Systein[56]. 


1.1  Engineering  Notes 

Building  MultiScheme  was  an  exercise  in  both  research  and  nuts-and-bolts 
engineering.  For  people  who  are  interested  in  the  cnginet'ring  aspects  of 
jiarallel-prograinming  systems,  a  number  of  important  lessons  are  described 
throughout  this  repcjrt.  This  section  surveys  these  le.s.son.s  in  tlu'  hojK’s  that 
if  will  ah'i  t  interested  readers  to  points  that  might  otherwise  be  overlooked. 


1.1.  ENGINEERING  NO  TES 


11 


Garbage  Collection 

Perhaps  the  most  interesting  engineering  result  comes  from  attempt¬ 
ing  to  implement  a  system  for  garbage  collecting  useless  tasks.  The 
intention  was  to  provide  support  for  speculative  computation  —  try¬ 
ing  a  number  of  alternative  ways  of  solving  the  same  problem  and 
selecting  the  result  of  the  first  method  to  succeed.  The  idea  of  ex¬ 
tending  the  garbage  collector  to  collect  useless  tasks  is  not  new  (see, 
for  example,  [9]),  but  to  our  knowledge,  MultiScheme  provides  the 
first  working  implementation  of  this  idea.  In  the  process  of  building 
such  a  garbage  collector  several  small  discoveries  were  made.  Most 
importantly,  the  choice  of  the  root  for  garbage  collection  is  critical  to 
the  success  of  such  a  system  (Section  3.5).  MultiScheme  uses  a  root 
that  combines  the  global  name  space  (the  ordinary  root  for  garbage 
collection)  and  those  tasks  created  explicitly  to  report  a  value  back 
to  the  user  (i.e.  tasks  running  the  standard  read-eval-print  loop). 

Furthermore,  the  data  structures  of  the  system  must  be  carefully 
crafted  so  that  the  garbage  collector  “sees”  only  tasks  that  are  per¬ 
forming  useful  work.  In  MultiScheme,  this  required  the  introduction 
of  weak  cons  cells  (Section  2.2.2)  to  allow  the  producer  of  a  value  to 
locate  the  consumers  of  that  value  without  retaining  the  consumers 
after  a  garbage  collection.  Even  with  all  of  this  support  in  place,  the 
garbage  collector  doesn’t  always  “do  the  right  thing”  since  some  tasks 
operate  by  side-effect  and  I  have  not  found  a  mechanism  for  detecting 
when  they  are  no  longer  needed  (see  the  example  of  Section  5.3). 

One  additional  result  related  to  garbage  collection  deserves  mention 
here  but  will  not  be  described  in  the  body  of  the  report  itself.  A  pair 
of  memory  management  strategies  were  developed.  Each  of  these 
includes  a  parallel  stop-and-copy  garbage  collection  algorithm  along 
with  related  decisions  dealing  with  the  mechanism  for  distributing 
the  physical  memory  across  the  shared  address  space  and  the  area 
of  memiory  available  to  each  processor  for  allocating  to  user  tasks. 
Tlie  initial  MultiScheme  memory  management  system  is  described 
by  Courtemanche[14],  including  details  of  the  garbage  collection  al¬ 
gorithm  and  measurements  of  its  performance.  Based  on  this  work, 
J(jhn.son[34]  wrote  and  tested  a  different  memory  management  sys¬ 
tem  designed  to  repair  some  of  the  problems  encountered  by  Courte- 
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manche. 

Weak  Cons  Cells  and  Object  Finalization 

Weak  cons  cells  (cons  cells  whose  car  becomes  ’  ()  rather  than  retain 
an  object  no  longer  needed  elsewhere)  and  object  finalization  (retain¬ 
ing  an  object  for  one  garbage  collection  cycle  after  it  would  normally 
disappear),  or  similar  mechanisms,  have  long  been  known  to  Lisp  im¬ 
plementors.  They  have  been  employed  internally  in  Lisp  systems  for 
at  least  twenty  years.  The  particular  form  they  take  in  MuItiScheme 
has  two  unusual  properties:  it  is  safe  enough  for  use  by  “application 
programmers”  who  have  little  knowledge  of  the  internals  of  the  Lisp 
implementation,  and  its  implementation  consisting  of  a  modification 
to  the  garbage  collector  and  a  post-garbage  collection  update  pass 
is  clearly  documented  (in  this  report).  Similarly,  the  notion  of  ob¬ 
ject  finalization  (Section  2.2.3)  has  been  around  the  implementation 
community  for  a  long  time.  Again,  the  implementation  described 
here  is  safe,  and  the  use  of  an  additional  garbage-collection  root  to 
accomplish  the  goal  is  described  here. 

Processor  Synchronization 

The  author  was  surprised  by  the  discovery  that  MuItiScheme  needed 
a  pair  of  low-level  operations  for  synchronizing  processing  elements. 
This  is  true  even  though  MuItiScheme  presents  no  notion  of  individual 
processing  elements  to  the  user.  However,  operations  such  as  starting 
a  garbage  collection  flip  (see  Section  3.1.1),  pausing  all  tasks  when  an 
error  is  encountered  (Section  3.1.3),  and  interrupting  a  task  that  is 
currently  running  on  another  processor  (Section  3.1.4)  inherently  re¬ 
quire  the  cooperation  of  the  processors  in  the  system.  The  particular 
choice  of  synchronization  operations  (synchronizers  and  global  inter¬ 
rupts)  is  not  profound,  but  the  fact  that  even  this  simple  set  provides 
an  adeciuate  base  for  implementation  of  the  higher-level  constructs 
demonstrates  that  only  small  extensions  are  needed  to  convert  Scheme 
into  a  convenient  systems  programming  base  for  a  parallel  processor. 

Error  and  Exception  Handling 

When  Hanson  and  Lamping[28]  showed  how  Stallman’s  dynamic- 
wind  operation[52]  could  be  seen  as  a  convenient  packaging  for  the 
two  independent  notions  of  a  dynamic-state  space  and  a  control-state 
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space  (continuations),  they  argued  that  this  provides  the  base  needed 
for  both  error  and  exception  handling.  In  extending  Scheme  into  a 
parallel-processing  environment  it  was  important  to  understand  the 
ramifications  this  would  have  on  these  same  issues.  Ajjpendix  D  docu¬ 
ments  the  analysis  of  this  area  and  provides  a  more  detailed  argument 
in  support  of  their  original  claim.  As  a  result  of  this  analysis,  no  new 
exception-handling  facility  was  added  to  MultiScheme.  Instead,  the 
dynamic-state  space  implementation  was  modified  to  operate  without 
side-effects  to  the  runtime  data  structures. 


1.2  Origin 


The  MultiScheme  effort  began  as  an  attempt  to  merge  the  best  parts  of 
two  existing  systems.  Loaiza  and  Halstead  had  developed  the  Multilisp 
system[26]  as  a  means  of  testing  ideas  about  parallelism  in  Lisp.  Their 
sj’stem  had  already  been  implemented  and  tested  on  the  MIT  Concert 
parallel  processor  and  served  as  a  vehicle  for  ongoing  research  into  parallel 
processing. 

In  addition,  a  group  of  hackers  had  developed  MIT  Scheme,  a  system 
used  in  teaching  undergraduate  courses.  Led  by  Chris  Hanson,  this  group 
was  constantly  revising  and  refining  both  the  local  Scheme  language  dialect 
and  the  system  implementation.  Chris  is  responsible  for  a  large  number  of 
the  system  ideas  that  ultimately  made  their  way  into  MultiScheme.  His 
gcniiis  for  taking  a  passing  comment,  studying  it,  and  producing  a  beauti¬ 
fully  worked  out  and  elegantly  abstracted  implementation  of  the  key  idea 
is  a  truly  rare  gift. 

In  attempting  to  integrate  the  ideas  from  these  two  systems  the  ad¬ 
vice.  help,  and  support  of  Guillermo  (Jinx)  Rozas  was  invaluable.  Jinx  has 
contributed  to  the  MultiScheme  effort  a  wide  variety  of  ideas  at  all  lev¬ 
els.  from  implementation  details  to  overall  system  structures.  His  untiring 
efforts  and  his  blasted  perfectionism  have  helped  move  both  MIT  Scheme 
and  MultiScheme  from  toys  to  rol)ust.  reliable  systems. 
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1.3  Initial  Conditions 

B\'  \Vc\y  (if  ba('k!2;i(miul  to  the  reinaiiider  of  this  ic-port,  this  section  serves 
to  position  the  MnltiScheine  project  with  resp('ct  to  other  work  in  the  field 
of  parallel  computation.  Although  it  is  tempting  to  provicU'  an  overvi(>w  of 
tlnit  entire  field,  th(‘  tlamie  of  incn'ased  computational  speed  through  the 
use  of  nmltiph'  computation  units  has  (“xcited  so  much  work  in  the  last  fi'.'e 
}vars  that  it  would  take  ser-eral  hooks  to  j>ro\'ide  such  an  ovf'iview.  The 
sur\'ey  hook  of  Filman  and  Friedman [20]  is  an  excelh'iit  introduction  to  the 
suhject . 

Tlu  strategy  a(lo[>ted  heiau  instead,  is  to  provide  a  description  of  the 
fi\'e  nuijor  houndary  conditions  within  which  the  work  Inis  proct'edi'd: 

Parallel  Computation 

MultiScheme.  from  the  outset,  was  intended  to  he  a  systimi  for  a  paral- 
li:l.  rather  than  a  di.*irthnt(:d,  computing  environment.  There  is  a  fine 
dividing  line  between  these  two;  and  the  distinction  I  draw  between 
th('m  centers  around  their  ability  to  .share  objects.  This  boundary 
condition  has  ramifications  for  computer  architectures  that  support 
MultiScheme,  and  the  matter  is  discussed  further  in  Section  1.3.1. 

Event-Driven  Computation 

The  primary  method  for  ensuring  precedence  constraints  between  user 
tasks  is  based  on  an  event-driven  ineclnmism.  Section  1.3.2  compares 
this  mechanism  to  two  common  alternatives,  busy  waiting  and  pollini,. 
The  twin  acts  of  touching  a  placeholder  and  providing  a  value  for  a 
placeholder  are  the  means  used  to  express  precedence  constraints  and 
thereby  realize  the  event-driven  model  in  MultiScheme. 

Explicit  Parallelism 

MultiScheme  was  intended  as  a  base  for  a  wide  range  of  experiments 
in  parallel  programming  methods.  As  such,  it  allows  the  programmer 
to  indicate  explicitly  how  the  parallelism  of  the  underlying  hardware 
is  to  be  exploited.  This  point  is  elaborated  upon  in  Section  1.3.3. 

.Automated  Resource  Allocation 

Writing  [larallel  programs  is  a  difficult  task.  MultiScheme  pro\'id('s  a 
default  mechanism  tluit  :mtomates  resource  tdlocation.  making  the 
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programmer's  job  significantly  easier.  MultiScheme's  approach  to 
processor  allocation  is  discussed  in  Section  1.3.4. 

Scheme  as  a  Base  Language 

The  Scheme  dialect  of  Lisp  was  chosen  as  a  base  language  for  Multi 
Scheme  because  of  its  conceptual  simplicity.  This  simplicity  allows 
a  larg(>  number  of  sequential-programming  models  to  Ire  ex])re.ssed 
within  the  language;  MultiScheme  is  intended  to  extend  this  expres¬ 
sive  power  into  the  parallel-programming  domain.  This  point  is  dis¬ 
cussed  in  Section  1.3.5 

1.3.1  Parallel  Computation 

MultiScheme  was  intended  from  the  outset  to  run  in  an  environment  con¬ 
sisting  of  many  computation  units  operating  independently  but  efficiently 
sharing  access  to  a  set  of  (mutable)  objects.  As  a  result,  the  programmers' 
model  of  the  system  need  not  include  the  notion  of  explicit  coinrnunica- 
tion  la'tween  tasks.  Rather,  the  programmer  thinks  about  objects  and 
op(>rations  on  objects  without  an  externally  imposed  constraint  that  ('ach 
object  have  a  jrarticular  task  or  processor  as  its  “owner.”  I  take  these  to 
be  the  distinguishing  features  of  a  parallel  system:  in  a  distributed  systc'm 
rh(>  programmer’s  paradigm  is  one  of  communication  between  tasks,  with  a 
concfunitant  notion  of  copying;  in  a  parallel  system  the  paradigm  is  sharing 
betwec'ii  potentially  concurrent  computations. 

Fundamentally,  this  is  a  constraint  on  the  class  of  architectures  con¬ 
ducive  to  a  MultiScheme  implementation.  An  architecture  must  i)ossess 
two  critical  features  in  order  to  make  an  acceptable  implementation  Irase: 

1.  A  large  portion  of  the  address  space  must  be  shared  by  (accessilde 
to)  all  processors  in  the  system. 

2.  The  speed  of  reference  to  all  portions  of  the  address  space  must  l)e 
comiiarable. 

Oil!'  (if  the  other  initial  conditions  is  the  choice  of  tlu'  Scheme  language 
'(■(•  Section  1.3.5).  Scheme  and  other  Lisps  siqiport  a  large  variety  of 
-  t  cl;i, ,  ol, ji'cf s  by  emjdoying  pointers  to  obji'cts.  rather  than  t he  ol i jects 
■  ;.e;;i'(T. cs.  as  the  fundamental  units  of  data  transh'r.  .Any  |)rocedure  with 
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to  a  [xaiitfi'.  <'itlicr  by  rt'c-riviug  it  as  ;m  tifftiiincut  or  by  aia-css 

Hi”  It  a>  the  N’aluc  of  a  laxiaally  vlsibl*'  variable,  must  li;i\’e  coinplete  access 
to  the  objeet  itselt.  But  elHeieiit  inii)leiiientation  of  this,  in  turn,  recjuiics  a 
il.it  aiMie>'  --jiace  fontainiinr  till  refi-r<’neeal)le  olijects.  It  is  a  nnitter  ot  folk 
w'l'il'  'in  in  tin'  Lisii  eoininuiiity  tli.at  iiuphuiientiitions  of  Lisp  on  seft,niente(l 
an'liltectuie^  ‘like  the  Intel  SOSO)  sutfer  siaiiificiiut  perforinaiu'e  penalties 
un!'-"  a  --in‘j,le  segment  is  larit.e  enouf^h  to  hold  the  entire  hetij)  or  the  Inard- 
h'aie  sMp[io;t>  the  ability  to  der<'ferene<'  a  pointer  to  any  location  in  the 
heap  lU'  a  !  as  in  the  Multics  Maclisp  inipleiuentitt ion ). 

( a  parallel  proce>^or,  this  saiiie  problem  apjauirs  in  :i  slightly  different 
irih'''  1  he  liui'j.uage  still  te(pures  a  flat  reference  spa<'e.  with  no  distinction 
bet  '.vei "a  objects  residing  "loi'al  "  to  :i  jirocessor  tuul  t hose  residing  '■remote  ' 
tiohi  the  processor;  and  ;\g;iin.  an  tirchitecture  that  impirses  In-tivy  peindties 
for  "remote"  address  references  will  l(>ad  to  infmior  [lerfornrance.  Moreover, 
ill  c;ius.‘  so  unuiy  kinds  of  oirjects  in  .Sclieiiu’  are  first-cl;i.s.s.  tin*  jjossildlity 
of  analyzing  progrtiins  to  divide  where  to  alloctiti'  objects  for  nuixinmin  lo¬ 
cality  of  reference  seems  tinlikely  to  stuveed  in  geuertil.  Thus,  ('(pially  raiiid 
access  to  till  objivts  is  re<[uired  for  stick  a  system  to  have  any  likelihood  of 
success. 

SIhtred  tuMress  spact^  luachiiies',  such  as  the  DDX  Dutterfiy[50].  MIT 
Concert [27j.  and  IBM  RP3[47]  systems  come  close  to  satisfying  the  ttbove 
criteria.  They  have  both  "local"  and  "remote”  memory,  but  the  hardware 
is  fully  responsible  for  dt'tecting  references  to  each  and  rt'acting  jiccord- 
ingly.  As  a  result,  on  the  largest  such  systems  (roughly  1000  jirocessors), 
local  memory  is  no  more  than  one  order  of  magnitude  faster  to  reference 
than  rt'uiote  memory  including  distinguishing  the  two  forms  of  rofmence. 
By  contrast,  the  Intel  iPSC  is  estimated  to  have  a  two-  to  three-ordcr- 
of magnitude  difference  in  reference  speeds  liccausi'  software  detection  of 
remote  rc’ferences  is  r('(|uired. 

This  difference  in  relative  access  speeds  is  e.xjjected  to  be  significant 
in  a  parallel  euvironuu’nt  as  definerl  here.  Since  the  iirogramming  model 
does  not  include  the  notion  of  local  and  remote  objects  corresponding  to 

'Siiiiv  tlif  .start  rif  tlic  NhiltiScficme  project,  a  class  of  machines  known  as  "message 
passing  macliines"  luis  been  [)ropose‘l,  of  wliicli  tlie  shared  addre.ss  machines  are  a  spe- 
‘  lal  ca.se.  'I'lie.se  machiiie.s  spun  a  wide  range  of  architect nres,  and  tliose  with  memory 
a‘-‘ ess  times  that  mei  i  the  two  criteria  mentioned  above  form  an  a[)propriate  ba.se  for 
M  nit  iS'heme 
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the  actual  nicinoiy  ronfiifiuation,  it  is  reasonable  to  consider  how  t';ese 
'y>r('iiis  perform  under  conditions  of  nniforinly  distributed  access  patterns 
and  (Worse  yet)  totally  remote  access  operations.  Under  these  kinds  of 
eoiiditions"  th('  eff('ctive  number  of  processors  is  reduced  linearly  by  the 
ratio  of  local  to  remote  access  speeds.  Thus,  a  Butterfly  (or  Concert  or  RP3) 
with  1000  physical  jtrocessors  should  be  capable  of  performing  roughly  two 
ordi  is  ()f-inagiiitude  bister  than  a  single  processor  from  which  the  system 
i'  made.  An  iPSC  of  the  same  size  could  be  expected  to  perform  at  best 
one  order-of-magnitvide  bister. 

In  addition  to  these  shared  memory  architectures,  there  are  two  addi¬ 
tional  mai'hines  tluit  satisfy  the  two  criteria  listed  above:  the  original  design 
ol  the  Connection  Machine,  tiiid  the  Monarch  computer.  The  original  pro- 
lio-^al  for  the  Connection  Machine[32]  had  a  large  number  of  processors, 
eaeh  with  ii  very  small  amount  of  local  state,  interconnected  by  a  sophis¬ 
ticated  routing  network.  From  a  programmer’s  point  of  view  the  machine 
ap[)eared  to  have  ;i  very  large  address  space  and  most  of  it  was  equally  ac- 
.•e>-ible  with  only  <'t  small  amount  of  state  information  more  easily  retiched. 
L'nfortuiuitely.  the  fact  that  the  machine  had  a  single  instruction  stream 
mtdtes  it  iiicomptitible  with  the  system  design  discussed  here. 

The  BBX  Monarch[ll]  has  a  large  address  space  supported  solely  at 
the  remote  end  of  a  switching  network.  The  goal  is  to  make  the  memory 
apjiear  to  be  a  single,  thousand-way  interleav'ed,  memory  unit.  In  addi¬ 
tion.  the  system  supports  tagged  data  making  it  extremely  well  suited  to 
the  implementtition  of  a  MultiScheme  system  as  described  in  this  report. 
I  nfortuiuitely.  the  ma  'hiue  is  still  in  the  design  stages. 


1.3.2  Event-Driven  Computation 

rile  ability  to  synchronize  tasks  is  extremely  important  in  any  kind  of 
multi  tasking  system,  and  it  becomes  more  important  as  the  interaction 
III  tween  tasks  b<-come.>  tighter.  In  a  parallel  processing  system  such  as 
the  one  envisioned  for  MultiSch(’me  this  problem  is  at  the  he;irt  of  the 
'y'-tein  design.  There  are  three  common  synchronization  methodologies 
leading  to  three  \-ery  different  kinds  of  system  structures:  busy-w;iiting. 
polling,  and  event  driven.  MtiltiScheme,  like  most  real  systems,  uses  till 

’.In't  linw  ;io  iir.itc  llicsf-  a.ssNin<-(l  acress  patterns  turn  out  to  he  in  prartin-  is  siihjeci 
I  i  nc  I'lir'  iii'  iit  for  any  given  systein  Nulli[-1.3]  descrihes  one  investigation  into  tins  ansa 


IS 


CHAPTER  1.  INTRODUCTION 


ti'i'lniicnu's  but  the  emphasis  at  the  language  (as  opposed  to  system) 

h'vel  has  been  placed  on  the  third. 

Busy-waiting 

Busy-waiting  is  a  very  simple  mechanism  used  to  serialize  access  to  a 
critical  resource.  There  is  a  lock  for  the  resource  and  a  task  repeatedly 
tries  to  acquire  the  lock  in  order  to  obtain  access  to  the  resource.  This 
organization  is  useful  when  the  likelihood  of  contention  for  a  lock  is 
very  small  and  the  lock  is  released  rapidly.  MultiScheme  employs 
busy-waiting  for  some  operations  such  as  locking  a  placeholder  (see 
Section  4.2.1)  and  to  compensate  for  some  unfortunate  defects  of  the 
current  hardware  (it  lacks  atomic  32-bit  swap  operations).  In  most 
ca.ses  it  is  possible  to  include  a  lock  with  the  individual  object  to  be 
locked  so  contention  only  occurs  if  two  tasks  attempt  to  access  the 
stune  resource  simultaneously.  Where  this  is  not  possible  because  the 
numb<'r  of  objects  requiring  locks  is  large  (for  example,  it  is  possible  to 
atomically  swap  a  value  with  the  value  of  a  variable),  the  objects  are 
grouped  together  and  locks  are  provided  for  relatively  small  numbers 
of  objects.  In  addition,  of  course,  the  critical  regions  (where  the  lock 
is  hold  by  one  task)  are  kept  as  short  as  possible. 

Polling 

Polling  is  ti  more  sophisticated  system  structure  involving  a  periodic 
test  foi'  events  requiring  service.  This  is  a  common  technique  in  mi- 
crocoded  machine  architectures  for  implementing  interrupts.  It  is 
also  found  in  higher-level  software  to  allow  interrupting  of  poten¬ 
tially  long  loops  (for  example,  the  search  loop  in  many  editors  checks 
for  user  interrupts).  Unfortunately,  polling-based  systems,  such  as 
th(’  Plurit)us[45|,  are  difficult  to  write  and  maintain.  The  software 
is  forced  into  a  model  of  short  program  strips,  each  of  which  polls 
for  an  ev('nt.  services  that  event,  saves  its  new  state  so  that  it  can 
be  resuiiK'd,  and  then  returns  to  poll  again.  Writing  such  a  system 
with  an  eye  toward  efficiency  is  tricky;  information  ordinarily  avail¬ 
able  fioni  th(’  location  in  the  code  and  the  contents  of  the  stack  must 
be  converted  into  a  data  structure  used  to  dispatch  on  events.  This 
is  eqiii\alent  to  converting  the  program  into  a  finite  stat('  machine 
(with  random  access  memory),  a  representation  that  is  hard  tt)  l)oth 
understand  and  modify. 


1.3.  INITIAL  CONDITIONS 


19 


While  coiiteution  is  generally  less  of  a  problem  in  a  polling-based 
system  than  in  one  with  busy-waiting,  contention  for  the  next  avail¬ 
able  event  becomes  a  problem  as  the  number  of  processors  increases. 
MultiScheme  employs  polling  for  servicing  interrupt  conditions  and  an 
early  version  of  MultiScheme  dramatically  demonstrated  the  existc-nce 
of  this  form  of  contention  within  the  system.  Combining  networks[lC] 
Itrovide  ti  hardware  mechanism  for  removdng  this  contention,  but  the 
hardware  on  which  MultiScheme  was  running  did  not  support  this 
mechanism.  Instead,  a  software  solution  has  proven  to  be  quite  effec¬ 
tive:  rather  than  provide  one  centralized  location  polled  by  all  proces¬ 
sors,  MultiScheme  provides  a  location  for  each  processor.  Signalling 
an  interrupt  requires  either  choosing  to  alert  a  particular  processor  or 
writing  the  flag  separately  for  each  processor.  Signalling  all  proces¬ 
sors  is  clearly  much  slower,  but  it  happens  infrequently.  The  polling 
that  must  occur  often  to  reduce  interrupt  latency,  however,  no  longer 
generates  contention. 

Event-Driven 

•A.  third  organization  can  be  used  when  a  resource,  once  allocated,  is 
likely  to  be  in  use  for  a  relatively  large  amount  of  time.  In  this  case 
it  is  reasonable  to  expect  the  task  using  the  resource  to  alert  other 
tasks  when  the  resource  becomes  available.  Tasks  suspend  themselves 
and  release  the  processor  to  a  different  task  when  they  discover  that  a 
resource  they  need  is  unavailable.  It  is  this  system  organization  that 
both  Multilisp  and  MultiScheme  attempt  to  make  readily  available 
through  the  use  of  futures  and  placeholders,  respectively. 

One  common  implementation  of  an  event-driven  system  is  the  mailbox 
or  inbox  appr(xi(  h.  In  this  organization  (see,  for  example,  the  CMOS 
operating  system[3])  each  task  has  a  specific  location  (the  mailbox) 
where  it  can  be  notified  of  events  as  they  occur.  Tasks  poll  this  mail¬ 
box  but.  unlike  the  polling  organization,  they  are  suspended  if  no 
event  is  availalde  for  processing.  Two  variations  on  this  theme  are 
the-  ability  to  wait  for  a  specific  set  of  events  to  occur  and  the  abil- 
iry  to  have  multiple  mailboxes  (such  as  the  events  of  the  Chrysalis 
syst<'m[2]).  MultiScheme  differs  from  this  approach  by  augmenting 
the  explicit  polling  with  the  automatic  touch  supplied  for  placehold¬ 
ers  (see  Section  2.1.1)  and  by  using  an  outbox  rather  than  an  inbox. 
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Each  task  in  MultiScheine  h^ls  an  associated  placeholder,  the  goal  it 
is  trying  to  compn.te  (see  Section  3.4).  Thus  a  task  has  a  unique  loca¬ 
tion  where  it  can  store  a  result  but  it  can  await  the  result  computed 
l^y  any  other  task.  The  “events”  in  MultiScheine  are  occurrences  of 
the  determine!  operation  (see  Section  4.5)  used  to  store  a  result  into 
a  placeholder. 


1.3.3  Explicit  Parallelism 

The  MultiScheine  language  enables  programmers  to  annotate  programs,  in- 
ilicating  where  the  programs  are  permitted  to  execute  in  parallel  with  an 
existing  computation  task.  The  decision  to  make  parallel  execution  visi¬ 
ble  and  under  the  control  of  the  programmer  reflects  two  major  language 
design  concerns;  the  language  should  encompass  both  functional  and  non¬ 
functional  programming  styles,  and  it  must  also  provide  program  analysis 
tools  (themselves  MultiScheme  programs)  a  mechanism  to  express  oppor¬ 
tunities  for  parallelism. 

Scheme  includes  side-effecting  operations  designed  to  enhance  the  mod¬ 
ularity  of  many  common  programming  styles.  .A.s  a  result,  the  automated 
detection  of  potentially  parallel  computations  is  quite  difficult.  Katz[35] 
and  Knight  [36]  describe  rather  elaborate  architectures  designed  specifically 
to  make  such  a  system  feasible.  Rather  than  reejuire  this  form  of  sup¬ 
port  from  the  underlying  architecture,  MultiScheme  relies  on  the  program 
to  indicate  explicitly  where  opportunities  for  parallel  computation  can  be 
exi>loite(l  without  affecting  the  semantics  of  the  jirogram.  It  is  entirely  pos- 
>ible.  in  MultiScheme.  to  request  parallelism  in  a  manner  that  makes  the 
ri'-ult,^  inconsistent  with  the  serial  execution  of  a  program  --  a  reflection 
of  the  fact  tlnit  the  additional  expressive  power  must  be  used  with  care. 

Furthermore,  if  the  language  is  to  serve  as  the  output  language  of  auto- 
iiiat('cl  piogram  analysis  tools,  then  it  must  allow  those  tools  some  way  to 
expie>'.  the  opportunities  for  jiarallel  execution.  The  MultiScheme  project 
ha>  not  c'o'jited  any  such  tools,  but  Giay[25]  and  Wang[59]  have  undertaken 
■-uc!!  [iioJ<'crs  for  the  Multilisp  language.  Th<*  job  of  these  tools  is  to  make 
eiiuineeiing  t riidf-ofl's  between  the  time  re<iuireil  to  create,  dispatch,  and 
ki!!  ta.'ks  and  the  time  re<juire<l  to  comj)lete  the  comjnitation  represented 
by  a  single  task. 
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1.3.4  Automated  Resource  Allocation 

A'  ;i  sy.'tciii.  MultiSchc'iiK*  sui)i)c)rts  automated  scheduling  of  tasks  onto 
oM'cessois  as  well  as  tli<'  detection  and  removal  of  no  longer  useful  tasks. 
The  underlying  system  supj)ort  for  scheduling,  discussed  in  Cha])ter  4.  is 
designed  to  allow  users  a  large  degree  of  flexibility  in  controlling  tin'  schedid- 
ing  policy  The  problem  of  programming  a  midti-processor  can  be  divided 
into  two  parts:  sutulividing  a  problem  into  components  for  parallel  ex('cm 
rion.  and  sehediding  these  components  onto  the  available  ])rocessor.s.  The 
e;;;:>licit  parallelism  constructs  support  this  first  task,  and  the  automated 
re-.()urce  management  sui>ports  th<‘  second.  Isolating  these-  two  components 
L,i^■e■-  programmers  the  freedom  to  de-sign  ane:l  te.st  the-ir  itrograms  inde-- 
pendent  of  the-  re-soure'e-  alleecatiem  e‘on.siderations  by  relying  e:)n  the-  de-fault 
allocation  strategie-s  of  the  standard  scheduler.  Later,  when  the-se  alloca- 
rioii  strategie-s  l)ecome-  a  j)erformance  concern,  the  scheduler  can  be  me)lde-el 
to  fit  the  particular  application. 

In  adelitie)!!  to  a  fle-xible  se-heduler  for  handling  allocation  of  processors 
to  tasks.  MvdtiSche-me  extends  the  Lisp  garbage  collector  to  remove  useless 
tasks  in  addition  to  its  traditional  role  of  recycling  memory.  Tl.is  allowed 
rill’  MultiSch<-me  work  ter  explore  the  area  of  demand-driven  computation 
within  Lisj),  tis  discussed  in  Section  3.4. 

1.3.5  Scheme  as  a  Base  Language 

(Tur  invi-stigation  into  parallel-jrrogramming  systems  might  have  Iregun  by 
'la-  e/r  tmeo  construction  of  a  hmguage  for  ewpre-ssing  ptu'tdlel  comi>utatie)ns. 
lu'tead,  we  chose-  to  u.se-  an  existing  language.  Scheme  (a  dialect  of  Lisj)). 
;is  our  luise-.  .A  number  erf  important  considertitierns  went  inter  this  choie'e; 

•  Scheme-  is  tin  e.xcellent  medium  in  which  ter  e-xireriment  with  ti  wiele- 
i.iiigi-  of  se-fiuential-programming  methodolergies  (se-e.  for  e-xtunjile-. 
.Abelson  anil  Sussman[7]).  .4n  important  geral  of  MultiSche-me  is  to 
jirovide-  this  same  rich  medium  ferr  experimentation  in  the-  jiarallel- 
progrti mining  domain. 

•  I.i'ps.  and  .Sche-me-  in  jrartie’ular,  le-nd  the-mseh’i-s  to  a  hirgi-ly  fune-- 
ilonal  --tyk  of  programming.  Thus,  the-  introeluction  of  ptiralli-lism 
m'o  ia  uii-  could  be  i-xpecti-el  to  Ire-  ;i  strtughtforwiird  I'haugi-  yieltl- 
u.'j,  a  'j,ood  deal  of  mcrease-d  speed. 
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•  The  Lisp  community’s  staple  domains,  symbolic  computation  and  ar¬ 
tificial  intelligence,  have  been  relatively  unaffected  by  the  availability 
of  hardware  capable  of  extremely  high  speed  (even  parallel)  arithmetic 
computations.  Parallel  processing  appears  to  be  a  good  candidate  for 
speeding  up  computations  in  these  domains. 

•  In  comparison  to  CommonLisp  (the  other  likely  candidate  Lisp  sys¬ 
tem),  Scheme  is  a  simpler  language,  with  far  fewer  special  forms  and 
required  procedures.  Furthermore,  because  of  its  first-class  continu¬ 
ation  objects  (see  Section  A. 4),  Scheme  has  a  more  powerful  control 
structure. 

•  A  language  extension  for  parallelism  (the  futures  of  Multilisp,  which 
became  the  placeholders  of  MultiScheme)  consistent  with  Scheme  had 
already  been  developed  and  demonstrated.  A  trio  of  alternative  con¬ 
structs  I  the  qlambda,  qlet,  and  qcatch  of  QLisp[23])  had  been  ana¬ 
lyzed  and  was  readily  implemented  in  MultiScheme,  suggesting  that 
the  placeholders  were  an  important  data  structure  in  any  case. 


1.4  Overview  of  the  Thesis 

The  main  l)ody  of  this  document  is  structured  to  reflect  the  three  major 
topics  mentioned  at  the  outset:  extensions  to  the  Scheme  language  (Chap¬ 
ters  2  and  3),  the  use  of  these  extensions  within  the  sequential  language 
(second  half  of  Chapter  2),  and  their  use  within  the  parallel-programming 
domain  (Chapters  4  and  5). 

Chapter  2  discusses  the  extended  sequential  Scheme  language,  derived 
from  the  MIT  Scheme  system  by  adding  placeholders  to  represent  uncom¬ 
puted  valaes  cuid  modifying  the  garbage  collector  to  provide  .several  services 
directly  visible  to  the  programmer.  The  latter  half  of  the  chapter  consists  of 
three  examples,  each  of  which  uses  this  extended  language  to  demonstrate 
a  solution  to  some  problem  of  current  interest  to  the  Lisp  and  functional 
languages  community.  These  examples  cover  a  range  of  topics:  embedding 
logic  variables  in  Lisp  (Section  2.3),  combining  normal  and  applicative  order 
evaluatic)!!  (Section  2.4).  and  higher-order  and  non-deterministic  streams 
processing  (Section  2.5). 
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Cha])ters  3  and  4  demonstrate  that  extended  sequential  Scheme  is  very 
dost'  to  a  language  suitable  for  systems  programming  of  parallel  processors. 
Theexten  sions  neetled  for  creating  this  systems-programming  language  are 
ili'sfiibed  primarily  in  Chapter  3.  They  comprise  two  distinct  changt's. 

•  The  at  blit  ion  of  a  few  new  primitive  operations  for  processor  coordi- 
mitioii  (Section  3.1).  task  distribution,  and  miscellanetnis  processor- 
oriented  operations  (both  in  Section  3.2). 

•  The  re-implementation  of  two  existing  mechanisms  for  use  on  a  paral¬ 
lel  ]U'ocessor.  The  first  mechanism,  fluid  variables,  is  used  to  provide 
private  storage  for  tasks  (Section  3.3).  The  second,  dynamic  state 
spaces,  provides  hooks  for  users  to  write  exception  and  error  han¬ 
dling  facilities  (Appendix  D). 

In  order  to  demonstrate  that  the  resulting  language  does,  in  fact,  make  a 
gcKjd  systems  programming  language.  Chapter  4  documents  the  implemen¬ 
tation  of  the  MultiScheme  scheduler  in  detail.  The  scheduler  is  responsible 
for  creating,  swapping,  and  terminating  tasks. 

Chapter  5  concentrates  on  expressiveness  of  the  MultiScheme  system. 
It  provides  three  diff’erent  example  programs  implemented  in  MultiScheme. 
The  first  of  these,  a  simple  rule-based  system,  demonstrates  how  Multi- 
Scheme  can  be  used  for  speculative  computation.  The  second  is  a  program 
to  solve  the  n-body  problem  of  classical  mechanics,  derived  from  a  serial 
program  by  the  addition  of  future  constructs.  The  third  is  a  more  elabo¬ 
rate  example  showing  how  a  vector  pipeline  similar  to  a  dynamic  dataflow 
programming  style  can  be  used  to  solve  the  same  ti-body  problem.  These 
examples  demon.strate  a  variety  of  ways  in  which  MultiScheme’s  facilities 
can  be  combined  to  provide  prototype  implementations  in  different  i)arallel 
programming  paradigms.  Each  example  is  fully  worked  out,  and  measure- 
iiK.’nts  of  its  performance  relative  to  the  serial  version  arc  supi)lied. 

In  addition  to  the  main  Ixjdy  of  the  report,  there  are  three-  ttpiiendice's. 
The  first  of  these  contains  background  material  for  readers  who  may  not 
be  familiar  with  the  MIT  Scheme  system.  The  text  of  the  report  assumes 
that  the  reader  is  familiar  with  the  details  of  the  standard  Scheme  language 
elc'scribed  in  [49]  and  the  MU'  ext  ensions  to  that  language.  .Ajipendix  .\  is 
intended  for  readers  who  tire  either  somewlnit  rusty  in  their  knowledge  of 
Lisp  or  who  are  familiar  with  ;i  different  dialect  or  implementation. 


24 


CHAPTER  1.  INTRODUCTION 


Appendix  B  coiitaiiis  the  iiapleiucntatioii  details  of  the  dataflow  (vector 
pipehiH')  s\'st('!n  described  as  part  of  the  example  of  Section  5.3.  It  is 
sn]ij)lied  for  those  readers  who  are  interested  in  understanding  how  such 
a  radically  different  programming  model  can  be  implemented  using  the 
facilities  of  MultiScheine. 

The  final  appendix  contains  a  variety  of  performance  measurements 
taken  from  a  number  of  different  versions  of  the  MultiScheine  system. 

1.5  Summary 

The  MultiScheine  work  centers  around  three  major  topics; 

1.  A  few  extensions  were  made  to  the  Scheme  language  in  order  to  pro¬ 
vide  a  !)ase  for  experiments  in  parallel  processing.  The  most  impor¬ 
tant  of  these  are; 

•  A  new  data  type,  placeholders,  can  be  used  to  represent  values 
not  yet  computed  (Section  2.1). 

•  The  garbage  collector  provides  services  that  are  directly  visible 
to  programs  (Section  2.2). 

•  Primitive  are  provided  for  coordinating  the  activities  of  the  sys¬ 
tem's  processors  (Section  3.1). 

2.  This  extended  Scheme  language  exhibits  new  expressive  power  for 
seritd  computations.  Sections  2.3,  2.4,  and  2.5  provide  illustrative 
examides. 

3.  The  ext('nded  language  provides  a  base  for  implementing  a  range  of 
parallebprogramniing  methodologies  (Chapter  5).  It  also  serves  as  an 
eh’gant  systems  programming  language  for  parallel  processing  (Chap¬ 
ter  4). 

In  the  j)rocess  of  building  MultiScheine  a  number  of  interesting  engineer¬ 
ing  residf'^  were  fliscovered.  These  discoveries  center  around  four  topics, 
'ununarized  in  Section  1.1  and  repeated  here; 

I  (iarbage  collection  (Section  3.4). 
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2.  Weak  cons  cells  and  finalization  (Section  2.2). 

3.  Processor  coordination  (Section  3.1). 

4.  Error  and  exception  handling  (Appendix  D). 

Section  1.3  describes  the  five  major  constraints  imposed  on  the  Multi- 
Scheme  project  at  the  outset: 

1.  The  area  of  exploration  is  parallel,  not  distributed,  systems  (Sec¬ 
tion  1.3.1).  This  requires  that  the  underlying  machine  support  an 
address  space  shared  across  all  of  the  processors  with  comparable 
access  times  to  all  parts  of  the  space. 

2.  Most  user  synchronization  is  based  on  an  event-driven  model  of  com¬ 
putation  (Section  1.3.2).  Programs  use  the  automatically  supplied 
touch  for  placeholders  to  augment  explicit  requests  for  synchroniza¬ 
tion.  The  determine!  operation  is  the  source  of  the  “events”  in  the 
system.  Busy  waiting  and  event  polling  are  also  employed. 

3.  User  programs  explicitly  specify  where  parallelism  is  to  be  employed 
(Section  1.3.3). 

4.  Users  do  not  deal  with  the  allocation  of  tasks  to  processors  (Sec¬ 
tion  1.3.4).  The  system  detects  and  removes  no  longer  useful  tasks 
and  supplies  a  structure  that  makes  it  straightforward  for  users  to 
specify  scheduling  policies. 

5.  Scheme,  a  dialect  of  Lisp,  serves  as  the  language  base.  The  MIT 
Scheme  system  serves  as  the  implementation  base.  (Section  1.3.5.) 
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This  chapter  describes  two  extensions  to  the  MIT  Scheme  system  that  pro¬ 
vide  additional  expressive  power  on  a  sequential  processor.  This  “Extended 
Sequential  Scheme”  is  the  base  from  which  MultiScheme  is  derived.  But  it 
is  an  interesting  language  in  its  own  right. 

The  first  extension,  described  in  Section  2.1,  is  the  addition  of  an  object 
to  represent  values  that  have  not  yet  been  computed.  These  placeholders 
are  similar  to  the  delayed  objects  of  standard  Scheme,  but  (unlike  thunks 
or  delayed  objects)  they  appear  to  be  replaced  by  the  value  they  represent 
once  it  has  been  computed.  They  also  form  the  implementation  base  on 
which  the  future  construct  of  Multilisp[26]  is  built  in  MultiScheme. 

The  second  extension,  described  in  Section  2.2,  is  more  of  a  change  in 
spirit.  In  standard  Scheme  the  garbage  collector  is  a  completely  invisible 
system  service.  Programmers  are  completely  unaware  of  its  operation,  and 
have  no  way  of  interacting  with  it.  In  extended  sequential  Scheme,  however, 
the  garbage  collector  can  be  used  as  an  active  partner  in  the  computation. 
The  addition  of  weak  cons  cells  allows  programmers  to  create  data  struc¬ 
tures  without  implying  that  the  objects  in  them  must  be  maintained  when 
the  garbage  collector  next  recycles  memory.  The  addition  of  a  user-specified 
garbage  collection  root  allows  users  to  ask  the  garbage  collector  to  indicate 
when  the  useful  lifetime  of  specific  objects  has  passed. 

The  remainder  of  the  chapter  uses  extended  sequential  Scheme  to  solve 
three  proldems.  These  problems  have  been  raised  by  other  researchers  and 
are  not  conveniently  solved  u.sing  the  initial  MIT  Scheme  system.  The  first 
problem  (Section  2.3)  is  the  embedding  of  logic  variables  in  Scheme  data 
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structures.  The  goal  is  to  allow  unification  of  expressions  containing  logic 
variables  to  produce  ordinary  Scheme  values  in  completely  independent 
data  structures  through  the  sharing  of  these  variables.  Since  a  placeholder 
becomes  the  object  it  represents  (once  its  value  is  known),  they  provide  a 
simple  solution  to  this  problem. 

The  second  problem  (Section  2.4)  is  the  introduction  of  limited  amounts 
of  normal  order  evaluation  into  an  otherwise  applicative  order  language.  By 
using  placeholders  for  selected  arguments  to  procedures,  a  very  fine  degree 
of  control  can  be  exercised  over  the  time  vs.  space  trade-off  inherent  in 
the  choice  of  evaluation  order.  Finally,  section  2.5  solves  a  collection  of 
difficulties  that  crop  up  when  dealing  with  stream  processing  in  either  an 
applicative  order  language  (higher-order  stream  functions  are  difficult  to 
write)  or  in  a  functional  language  (fair  merge  is  not  a  function).  The 
placeholder  is  a  convenient  mechanism  for  solving  both  of  these  problems. 

2.1  Placeholders:  A  New  Data  Type 

The  first  and  most  pervasive  extension  made  to  the  original  MIT  Scheme 
system  was  the  addition  of  a  new  data  type  with  some  unusual  proper¬ 
ties.  This  type,  for  historical  reasons  called  future  in  the  MultiSclu  me 
implementation’,  is  best  described  as  a  placeholder;  it  provides  a  way  to 
represent  an  object  whose  value  is  not  known  yet  or  may  be  subject  to 
change  later.  It  provides  a  form  of  sharing  difficult  to  obtain  without  di¬ 
rect  language  support;  the  same  object  can  occur  in  many  data  structures, 
but  tlie  actual  choice  of  object  can  be  altered  without  knowing  the  data 
structures  containing  the  object^. 

An  alternative  view  (more  related  to  the  implementation  than  to  lan¬ 
guage  design  goals)  is  to  consider  a  placeholder  as  an  “invisible  indirect 
reference”  t(r  another  object.  Yet  another  view  (helpful  in  explaining  to 
people  familiar  with  Scheme)  is  that  a  placeholder  is  very  similar  to  a  de¬ 
layed  object  (or  a  promise)  except  that  the  force  operaf  lequired  to  access 

'Fut'ires  were  described  by  Baker  and  Hewitt[9]  and  formed  the  beisis  for  the  iinple- 
nientation  of  the  Miiltilisp  .sy.stem  described  by  Halstead[26].  Placeholders  in  MultiSchenie 
are  directly  based  on  these  concepts  but  differ  in  a  number  of  ways. 

'.As  will  be  discussed  in  Section  2  .'1.  tliis  form  of  sharing  appears  to  be  the  critical  idea 
beliiiid  the  '’logic  variable"  of  Prolog.  This  form  of  variable,  in  turn,  provides  a  great  deal 
of  the  p(;\vi’r  ;iiid  flexibilits  of  the  logic  programming  languages. 
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the  value  is  performed  automatically  when  needed.  Accessing  the  value  of 
a  placeholder  is  called  touching  the  placeholder. 

This  last  view  points  up  two  important  notions.  The  first  is  that,  just 
as  in  stream  processing  in  Scheme,  the  placeholder  can  be  used  to  denot(“ 
an  object  whose  value  is  not  yet  known.  Thus  a  placeholder  object  has  at 
least  two  states.  A  placeholder  is  said  to  be  undetermined  when  the  ob¬ 
ject  it  references  is  not  yet  known.  Unlike  the  delayed  object,  however,  a 
placeholder  need  not  have  an  unchanging  value.  Thus,  a  placeholdi-r  ac¬ 
tually  has  three  states:  undetermined,  determined,  and  mutably  determined. 

mutably  determined  placeholder  may  have  the  object  to  wliich  it  reh'is 
changed  by  re-dettniuiuing  the  placeholder,  while  a  dctermiiu'd  placehohU'r 
cannot  change  its  vahu'.  Notice,  also,  that  delay  is  a  special  form  in  Scla'ine 
whose  evaluation  results  in  a  delayed  object.  The  placeholder  being  dis¬ 
cussed  lu'ie  is  a  data  type  and  not  a  special  form.  That  is,  “delay"  refers  to 
a  programming  construct,  while  “delayed  object”  and  "placeholder"  refer 
to  data  ol)jects  manipulated  by  programs. 

Since  a  placeholder  can  represent  an  object  before  the  object  even  ex¬ 
ists,  it  is  logical  to  ask  what  happens  when  a  program  touches  such  a 
placeholder.  This  is  controlled  by  the  scheduler,  a  portion  of  the  syst('m 
code  written  in  MultiScheme  and  hence  under  programmer  control.  This 
portion  of  the  scheduler  is  discussed  in  detail  in  Section  4.4,  but  in  general 
the  idea  is  that  the  task  performing  the  touch  is  suspended  until  the  value 
of  the  placeholder  has  been  determined. 

The  second  point  raised  by  the  comparison  with  Scheme’s  delayed  ol)- 
jt'crs.  explored  in  detail  in  Section  2.1.1,  is  the  fact  that  a  plac('holder  ob¬ 
ject  is  aatamaticalUj  touched  by  certain  primitive  operations  and  liinguage 
constructs.  In  stream  processing  it  is  important  that  the  use  of  force  to 
retrieve  the  value  of  a  delayed  object  be  deferred  as  long  as  possible.  This 
leads  to  a  "lazy  evaluator"  where  stream  processing  can  be  used  as  a  model 
for  call-by-need  computations.  The  question  of  when  an  object  can  remain 
a  placeholder  and  when  it  must  take  on  an  actual  value  is  the  topic  of  the 
next  subsection. 

The  schefluler  is  also  involved  in  the  creation  of  jilaceholder  olqects 
(as  described  in  Section  4.3).  It  determines  how  <md  when  coni])uting  re¬ 
sources  should  be  d<-\'oted  to  finding  tin'  valu('  of  the  ])laci'holder.  When 
the  ])laceholder  oljject  is  used  to  iini)lemeut  tin*  <'(juivalent  of  a  Scheme 
lielayefl  olqecf  no  resources  ar<“  allof'ated  at  the  tune  tlie  pIac('lioI(ler  is 
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I  icatt'd.  Instead,  they  are  allneati'd  when  it  is  touched.  Parallel  execn- 
tii'U  in  MnltiSeheine  is  clnsely  related  to  the  use  of  ])laeeh()lders.  :md  it 
is  largely  th('  iu'tion  of  the  seheduler  when  a  phteeliohler  is  created  that 
distinmiishes  stretun  procf'ssing  (lazy  e\-aluation)  from  ptirallel  j^roeessing 
i  eager  e\'alu;ition).  A  parallel  computation  is  initiated  by  creating  ;m  un¬ 
determined  phiceholder  to  contain  (eventinilly )  the  vidvie  returned  liy  tlnit 
comimttition.  The  scheduler  also  creates  a  task  to  perform  the  coinjtuta- 
tion  ;uid  {novides  it  a  continuation  that  ctiust's  the  t;isk  to  determine  the 
j)laceholdi'r  ;ind  then  terminate.  The  (undetermined)  phiceholder  is  thus 
immeilititely  availalde  to  the  initiating  task,  which  continues  its  computa¬ 
tion  e.xtictly  ;is  thi^ugh  it  had  received  the  fimd  vtilue  of  the  computiition. 
Meanwhile,  the  scheduler  has  released  the  task  for  i)otenti;il  simultaneous 
e.vecution  on  another  processor. 


2.1.1  What  Does  “Invisible”  Mean? 


In  order  to  allow  an  undetermined  placeholder  to  ;ict  as  an  invisible  place- 
holdi'r  ;is  long  <ls  possibh.-  (and,  therefore,  permit  as  much  paicdlelism  as 
jiossible)  it  is  necessary  to  define  a  set  of  rules  for  deciding  when  the  place¬ 
holder  c;m  be  usi'd  aiul  when  the  object  it  references  must  actually  be  ob- 
M’l'.ed.  A  similar  problem  ari.ses  whim  creating  a  norimil  order  int('rj)reter 
(t-.  c!i-t'n.-,sed  liy  .Abelson  <md  Sussman[7]  (section  4.2.1)  and  a  solution  sim¬ 
ilar  to  the  one  outlin<'<l  there  .solves  the  proltlem  very  nicely. 

The  imjiorfant  thing  to  notice  is  flnit  each  of  the  Scheme  sj>eci;d  forms 
can  be  a iialyzed  to  determine  whetlu'r  it  can  deal  with  placeholders  or  must 


bject  itsi’If  m  h;md.  The  Scheme  language  consists  only  of  tlu'se 
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.  f  ii  ni.'.  \  ariable  references  ;md  jirocedure  ctills.  Thus,  once  this  anal- 
I'aiiii-  th»'  pi  I  .j  )i’rt  ii"-  of  all  conipoumi  (i.e.  user-defineil )  proceilmes 
dii<  ctly  The  only  remaining  areas  tire  thus  the  prnniti\'i'  jiroci- 
't  Me  language  and  th<‘  somewhat  subtle  problems  arising  from  the 
.]■  Si  hena'  can  manipulate  its  programs  as  data  olijects.  The  results 
an.il;,  ato  suniinari/ed  l>elow.  E.ach  ca.si'  is  annotatecl  to  indii'ate 
I  f'a  tcuch  1"  j)io\ided  tiy  the  intei](reter  (or  coinjiiler).  or  is  the 
-;M;ht',  Ilf  the  codi-  for  some  primitive  procedures. 
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Where  to  Automatically  Toucli 

1.  Tlie  value  of  the  predicate  in  the  special  forms  if.  cond.  and.  or.  and 
case.  This  must  l)e  handled  by  the  interpreter  and/or  compiler. 

2.  The  value  of  the  operator  in  an  application.  This  must  be  handled 
by  the  interpreter  and/or  compiler. 

3.  An  expression  presented  for  direct  evaluation  (i.e.  in  the  case  di.spatch 
of  the  interpreter  itself  a  placeholder  object  will  be  touched  rather 
than  tn'ated  as  a  self-evaluating  constant).  This  must  be  handled  by 
the  interi)reter.  This  ca.se  does  not  arise  in  normal  practice,  although 
it  would  be  important  if  the  system  were  changed  to  allow  the  program 
constructors  (the  "syntaxer”  of  the  MIT  Scheme  system)  to  operate 
in  parallel. 

4.  The  value  of  operands  to  a  primitive  procedure  when  they  are  required 
to  be  of  a  specific  tyi)e  or  set  of  types,  as  well  as  any  portions  of  a  data 
structure  iiispei'ted  by  such  a  primitive,  unless  (of  course)  the  prim¬ 
itive  recpiires  a  placeholder.  This  must  be  handled  by  the  primitive 
itself.  For  example,  the  primitive  operator  +,  which  requires  numeric 
.•irgumeuts.  must  touch  each  of  those  arguments.  The  arguments  to 
th<'  cons  and  list  operations,  by  contrast,  may  be  of  any  type  and 
thus  need  not  be  touched. 

■'  The  value  of  oi)er;uKls  to  primitive  predicate  procedures  ((,‘xce])t  the 
predicates  future?  and  non-touching-eq?).  For  extimple,  the  j)rmi- 
iti\'e  operation  eq?  is  a  j)redicate  (it  is  guaranteed  to  return  eitluu' 
#T  or  #F  regardless  of  its  arguments).  This  must  be  handled  l)y  the 
jjiimitive  itself. 

C.  Certain  primitives  added  to  deal  with  plac(diolders  handh’  th('  touch 
o])erati(>u  exi)licitly.  For  ex;uni)le,  the  primitive  operation  touch  is 
pro\  ide(l  as  a  way  to  explicitly  touch  an  ol)ject  in  case  it  is  a  ])lace- 
holder.  This  must  be  handh'd  by  the  primitive  itself. 

.1.2  Example:  A  (Contrived)  Data  Base  Example 

an  examjile  of  the  kind  of  opertitions  j)ossible  with  i)laceholders  in  the 
uiguage.  roiisider  a  .Scheme  progrtim  used  to  miuntain  ti  data  base  of  stu- 
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(IriiT-  .  iiin’utly  ciirollccl  in  a  class.  Let  iis  assuiiu',  for  simplicity,  that  the 
.'.••'a  ha.^c  is  a  list  of  rccortls,  one  record  for  each  student.  The  records 
eia.f.sin  "unie  i  imunitahle)  data  fields  such  as  a  name  and  teacher  assip;m 
:.r.  W’e  also  want  to  store.  f(.)r  each  student,  the  f:,rade  on  each  (piiz. 

'  xani.  and  piolilem  set.  As  each  of  these  is  graded  we  ston'  the  grade  into 
rh'  appiayiiiate  fit'ld  of  the  record.  This  recpiires  fields  that  receive  their 
'.a.i  '.e  iHii  r  the  data  Itase  is  cr<’at('d  through  a  single-assignment  operatictn. 
K' a  -imp’.ieity.  wa-’ll  consider  only  one  such  number,  the  grade  on  the  fimd 
exam. 

'  far.  rlii-  is  an  easy  task  (if  we  ignore  the  usual  problems  of  efficiency, 
'.■na  ti  ;m  diarjv  stoiaage.  and  atomicity  of  simultaneous  actions).  All  that  is 
u-  ed.,  li  i-  tij  create  a  data  abstraction  for  th?  records  including  a  construc- 
oi;  1  Make-Student-Record),  selectors  for  each  component  of  the  recttrd 
'  Student  .  Name.  Student  .Teacher,  Student .  Grade),  and  a  writer  fc^r  the 
'ingle  assignment  entry  (Set-Student  .Grade ! ). 

Now  imagine  that  recitation  instructors  want  to  routinely  examine  the 
data  ba'C  to  see  certain  information  about  students  assigned  to  their  sec¬ 
tions.  They  should  be  able  to  create  structures  of  their  own  choosing  con¬ 
taining  parts  of  the  main  data  records.  For  example,  I  might  want  to  create 
a  li't  of  the  fimd  exam  grades  of  students  in  my  section.  As  the  main  data 
l;a.'e  i-,  tipdated  tifter  tlie  final  exam,  I  expect  my  list  to  reflect  tlie  final 
e.xam  grades  of  my  students.  When  the  data  base  is  created,  I  plan  to  tlo 
f  illowing:  (the  proc<'dure  filter  returns  a  list  of  items  from  an  input 
h't  that  match  a  selection  criterion) 

(define  My-Sect ion-Grades 
(map  Student . Grade 

(filter  All-Students 

(lambda  (Student-Record) 

(eq’  (Student .Teacher  Student-Record)  'JIM))))) 

I  nf"!  f  mi.itely,  thi'  ;iiipro;ich  dcx'sn't  work  if  we  use  the  “obvious"  im- 
;  T  n.' !.•  at ;i -n  of  the  d;ita  tdjstr.action.  Since  the  variable  My-Section- 
Graies  ted  when  the  d.ata  ba.se  is  initialized,  the  values  for  the  Stu- 

ier.*:  .  Grade  field  will  have  tiieir  original  (uninitialized)  value  and  this  vtdue 
.d  ■’  'c  npda'ed.  when  the  final  ex;un  grtide  is  entered  into  the  d;tta  b;\se. 
I  '..;  -  1-  pi't  one  in-tanc.'  of  whiit  Sussniiui  ;m<l  .Al)elson  refer  to  :is  "time  of 
■  fi. ,  e.  Tmii- of  construction";  we  would  like  M} -Section-Grades  to 
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have  its  component  values  computed  only  when  they  are  selected  from  the 
list,  not  when  the  list  is  constructed. 

There  are  a  number  of  ways,  in  standard  Scheme,  of  dealing  with 
the  problem  described  here.  Perhaps  the  simplest  comes  down  to  mak¬ 
ing  each  field  that  will  receive  a  value  after  the  data  base  is  constructed 
contain  a  (single-element)  list  with  the  value  of  the  field.  Then  this  list 
is  contained  in  My-Section-Grades,  and  the  list  is  mutated  by  the  Set- 
Student .  Grade !  proced  ire.  Unfortunately,  this  requires  that  we  think  of 
My-Section-Grades  as  a  list  of  lists  rather  than  as  a  list  of  numbers.  Thus, 
instead  of  the  straightforward  definition 

(define  Hy-Section-Average 

(/  (apply  +  My-Section-Grades) 

(length  My-Section-Grades))) 

we  are  forced  to  say 

(define  (Extract-Grade-From-Record  Record) 

(first  Record)) 

(define  My-Section-Average 
(/  (apply  + 

(map  Extract-Grade-From-Record  My-Section-Grades)) 

(length  My-Section-Grades))) 

In  essence,  we  are  forced  into  introducing  a  new  data  structure  for  the 
sole  jiurpose  of  providing  these  single-assignment  fields.  Furthermore,  the 
modularity  of  our  program  is  changed.  It  is  necessary  to  provide  support 
for  the  use  of  this  new  data  structure  for  grades  rather  than  using  the 
support  already  provided  for  handling  numbers.  As  the  comjilexity  of  the 
data  base  increases  by  the  addition  of  single-assignment  fields  or  other 
srructures  sharing  these  fields,  the  number  of  changes  needed  to  support 
the  late-arriving  data  also  increases. 

Once'  placeholders  have  been  added  to  the  language,  however,  a  more 
elegiiiit  solution  becomes  possible.  The  constructor  Make-Student-Record 
i-'  niodifi('d  so  that  the  initial  value  of  the  single-a.s.signment  field  is  an  unde- 
0-1  iiimcd  placeholder.  The  definitions  for  the  writers  of  the'  student  records 
iiinsf  use  Determine!  instead  of  Set!  to  store  the  new  value's.  Since'  the 
placi'heeleler  be'conie'.s  invisible  when  it  is  determine'd,  My-Section-Grades 
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acts  ('xtu'tly  like  a  list  of  imiubors  and  the  code  we  originally  wrote  for 
calculating  av('rages  works  as  we  ha<l  expected. 

But.  liy  using  tin*  fact  tlnit  placeholders  can  be  mutated,  we  can  solve  an 
lA'en  harder  yirobleiu.  Imagine  that,  in  addition  to  the  single-assigiinicnt 
tii'l(l>  with  ([ui/.  exam,  tu.td  proljlein  set  grades  we  want  to  maint:\iii  an 
estimated  coui'C  gr;ul<'  for  each  student.  This  number  should  be  updated 
a'  the  i;e\N-  grades  tire  entered,  and  thus  this  field  is  fully  mutable  rather 
than  singlf'-assignable.  By  using  the  mutable-determine !  (rather  than 
determine!  !  (!peration  to  ufulate  this  field,  we  can  allow  this  value  to  be 
>elected  by  iustrtictois  wh(‘n  the  data  base  is  constructed.  It,  to(^,  will  be 
updated  whenever  the  entry  in  the  main  data  base  is  updated. 

This  exaiuph'.  while  somewhat  contrived,  does  represent  an  important 
Contribution  of  the  placeholder  d;ita  type  to  Scheme.  The  kind  of  shar¬ 
in';  de.s.'ribed  here  coupled  with  the  invisibility  of  the  placeholder  object 
it  x'If  provides  the  tuuleilying  support  for  a  variety  of  capabilities  not  easily 
realized  in  the  standtird  Scheme  language.  Some  of  these  capabilities  are 
explored  in  si'ctious  2.3,  2.4,  and  2.5. 

On  the  other  In'ind.  of  course,  the  more  general  problem  of  data  base 
consistency  is  nut  soh'ed  by  the  use  of  phaceholder.  For  example,  the  auto¬ 
matic  touching  rules  listed  earlier  require  that  a  placeholder  be  determined 
liefore  a  decision  is  made  based  on  its  value.  Yet  if  the  value  of  that  place¬ 
holder  is  changed  there  is  no  mechanism  to  “unwind”  decisions  based  on 
the  previous  value.  Thus,  while  the  mechanism  supporting  the  placeholder 
construct  is  tpiite  powerful  and  flexible,  I  do  not  know  how  to  efficiently 
generalize  it  into  a  truth  maintenance  system. 

2.2  Talking  to  the  Garbage  Collector 

Scheme,  in  common  with  other  Lisp  dialects,  provides  operations  to  allocate 
-pace  fn  an  lueinory.  In  addition,  certain  operations  inherently  require  space 
O)  lojue-enr  objects  (>ven  when  these  objects  are  to  be  used  as  intermediate 
ii'-ult-,  .At  the  same  tiimn  Schemt'  does  not  provide  any  operations  for 
I'v  a-ing  this  memory.  Instead  it  relies  on  a  garbage  collector  (or  GC)  to 
n  1  'C.i  :  allocated  !)ut  uureferenceable  memory 

rii'  K  ha\'e  been  three  areas  where  the  MultiSclumie  project  has  rx- 
]<[•  nod  'nailiage  <-ollcction.  The  first  of  these  is  the  construction  of  two  juir- 
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allcl  stop-aml-copy  j>;arbage  collector  implementations.  The  earlier  of  thes(' 
implementations  is  described  by  Courtenianche[l4].  and  was  later  refined 
by  Kirk  Johnson  to  prodnce  the  second  implementation.  The  second  piece 
of  work  relates  to  the  garbage  collection  of  useless  tasks,  and  is  described 
in  Section  3.4. 

The  third  area,  described  in  this  section,  is  a  pair  of  extensions  (weak 
cons  cells  and  the  ability  to  discover  when  selected  objects  are  no  longer 
needed)  intended  to  allow  the  garbage  collector  to  act  as  a  partner  in  the 
computation.  These  extensions  allow  the  garbage  collector  to  provide  infor¬ 
mation  derived  during  its  pass  o%’er  the  entire  active  storage  of  the  system. 
In  order  to  accomplish  this,  however,  the  garbage  collector  can  no  longer 
promise  to  leave  the  state  of  the  system  apparently  unaltered  as  has  tradi¬ 
tionally  been  the  case  —  some  aspects  of  the  MultiScheme  garbage  collector 
are  deliberately  visible  to  users  of  the  system. 

The  actual  garbage  collector  used  by  MultiScheme  requires  that  the 
system  suspend  all  other  activities  while  it  is  garbage  collecting.  The 
changes  described  here,  however,  appear  to  be  compatible  with  the  ■‘real- 
time"  garbage  collection  algorithms  derived  from  the  work  of  Baker  and 
Hewitt[9].  In  a  system  with  a  real-time  garbage  collector  the  GC  demons 
described  below  would  run  at  the  time  when  semi-space  flip  occvirs. 

2.2.1  Garbage  Collection  Demons 

Prior  to  the  MultiScheme  project,  MIT  Scheme  provided  a  mechanism 
known  as  GC  demons  for  interacting  with  the  garbage  collector.  Users 
could  provide  procedures  that  would  run  immediately  after  each  garbage 
collection  and  before  returning  to  the  work  in  progress  when  the  gaibage 
colk'ction  began.  By  default,  the  system  supplied  two  demons  to  perform 
oi)eratif)ns  required  by  the  system  itself:  a  hash  demon  to  support  tables 
indexed  by  arbitrary  objects  and  a  files  demon  to  close  files  no  longer  in 
use. 

The  hash  demon  maintains  a  pair  of  tables  in  order  to  provide  a  form 
of  "unique  ID"  service  for  arbitrary  Scheme  objects.  An  object  can  be 
iuserfed  into  these  tables  using  the  primitive  object-hash,  which  returns  an 
integer  that  serves  as  a  unique  ID  for  that  object  (a  subsequent  attempt 
fo  insert  the  same  object  will  return  'he  same  unique  ID).  The  tables  can 
be  searched  using  object-unhash  which  expects  as  inp\it  this  uniejue  ID  and 
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rerunis  tlie  associated  object.  The  special  feature  of  these  tables,  recjuiriiig 
suppent  from  the  garbage  collector,  is  that  an  object  not  referenced  by 
any  other  Scheme  object  will  disappear  from  the  hash  tables.  After  each 
garbage  collection  the  tables  are  rebuilt  by  the  hash  demon  based  on  only 
those  objects  surviving  the  garbage  collection. 

The  files  demon  is  responsible  for  detecting  the  fact  that  a  file  is  no 
longer  referenceable  from  within  Scheme.  It  then  calls  on  the  operating 
system  to  close  the  file  if  the  user  hasn’t  already  done  so  explicitly.  This 
permits  files  to  be  handled  rather  like  memory  objects:  they  arc  created 
but  neetl  never  be  explicitly  released.  The  operation  is  supported  by  a  table 
of  tile  objects  hidden  from  the  GC  primitive.  This  table  is  rebuilt  by  the 
files  demon,  after  each  garbage  collection,  by  copying  entries  seen  during 
the  garbage  collection  or  by  closing  files  that  were  not  encountered. 

Each  of  these  demons  reejuires  the  ability  to  examine  the  portion  of  the 
address  space  from  which  o’ojects  had  been  copied  (old  space),  and  therefore 
could  not  safely  be  written  in  Scheme.  In  the  process  of  developing  Multi- 
Scheme  two  new  features  were  added,  supported  by  changes  to  the  garbage 
collector.  The  first  of  these,  weak  cons  cells  (discussed  in  Section  2.2.2), 
serves  as  an  implementation  base  for  a  new  hash  demon  that  does  not  need 
to  refei-ence  old  space.  Tliis  2jew  hash  deijion,  therefore,  is  written  entirely 
in  Si'heme.  Weak  cons  cells  also  serve  a  vital  role  in  MultiScheme’s  ability 
to  garbage  collect  useh-ss  tasks,  as  discussed  in  Section  3.4.  Weak  cons  cells 
ha\'e  --ince  been  used  to  perform  other  vital  system  services,  especially  in 
'Vg.ptiir  of  interfacing  to  compiled  code.  The  weak  cons  cells,  themselves, 
-'till  i<'(|uire  :i  demon  that  Icjoks  at  old  space,  but  it  is  far  less  complex  and 
vrr  >uppoi  ts  a  wider  range  of  services  than  the  original  hash  demon. 

Tht'  sec(md  feature,  object  finalization  (discussed  in  Section  2.2.3),  is 
of  ^onll’what  more  limited  use.  It  permits  the  files  demon  to  be  written  in 
Scheme'.  We  had  also  anticipated  that  it  would  provide  a  base  for  certain 
user  seiu'ices  in  MultiScheme.  These  services,  however,  have  never  been 
implemented. 

2.2.2  Weak  Pointers 

Th'-  tables  supijorted  by  tlu'  hash  demon  provide  two  independent  services 
in  olio  wraiJiier:  interning  and  weak  pointers.  The  interning  service  is  the 
ono  diio.-fly  visilde  from  its  interface  functions,  object-hash  and  object- 
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unhash.  Inteniiiig  provides  a  (reasonably  rapid)  method  for  generating 
a  unique  ID  for  an  arbitrary  object.  Since  the  unique  IDs  are  integers 
they  provide  an  ordering  on  Scheme  objects.  This  permits  ordered  data 
structures  to  be  constructed  from  arbitrary  Scheme  objects. 

The  second  service,  weak  pointers,  is  what  allows  objects  referenced  only 
by  way  of  the  interning  service  to  disappear  during  a  garbage  collection.  It  is 
the  vital  support  needed  for  maintaining  an  association  between  a  property 
and  the  set  of  objo^cts  having  that  property.  (Consider,  for  example,  the  set 
(jf  objects  that  a  user  has  traced  for  debugging  purposes.)  While  it  may  be 
necessary  to  provide  operations  referring  to  all  objects  in  the  system  having 
the  chosen  property,  it  is  not  intended  that  the  objects  remain  in  the  system 
for  the  sole  reason  that  they  have  that  property.  Maintaining  this  sort  of 
association  is  sufficiently  common  that  several  Scheme  dialects,  including 
MIT  Scheme,  provide  a  data  structure  for  handling  this  situation,  known 
as  a  population.  Empty  populations  can  be  created,  and  then  objects  can 
be  added  and  removed  from  them.  Operations  are  provided  for  applying 
a  function  to  each  remaining  element  of  a  population.  An  object  leaves  a 
population  either  by  an  explicit  user  operation  or  when  the  garbage  collector 
discovers  that  no  reference  to  it  exists  outside  of  populations. 

A  particularly  important  use  of  populations  in  MultiScheine  is  in  main¬ 
taining  the  collection  of  tasks  waiting  for  the  result  of  a  given  comi)utation. 
This  collection  is  conceptually  a  population  since  there  is  no  need  to  re¬ 
tain  a  task  just  Ijecause  it  is  currently  waiting  for  a  result  to  Ije  computed. 
Rc’presenting  this  collection  using  a  population  rather  than  a  traditional 
(unordered)  set  data  structure  is  what  allows  the  MultiScheine  garbage 
colh'ctor  to  quench  speculative  parallelism. 

Populations  can  be  built  on  top  of  the  tables  maintained  liy  the  ha.sh 
deiiK.ni.  This,  in  fact,  was  the  implementation  used  in  MIT  Scheme.  But 
nowhere  in  the  description  of  the  population  abstraction  did  it  become 
necessary  to  rely  on  the  ability  to  order  the  objects  within  tlu'  population 
Thus,  support  for  this  abstraction  uses  only  the  weak  pointer  servic<-  of  t!,!' 
rehash  demon,  not  its  interning  service.  Indei'd.  in  examining  tic  entm’ 
exi.sring  MIT  S<'heme  system  as  well  as  programs  written  in  Scheme.  nU 
uses  of  the  unique  IDs  recpiin'  only  the  weak  pointer  seiu'ice  ( although  me-, 
ior  the  inteniiug  are  still  jinticip.ated).  Mtiinttiining  the  tables  nsjuii  es  1,,  ,th 
time  and  space,  and  yet  they  could  be  <-omj>letel\’  omitted  if  MIT  Schi  ine 
had  support  for  weak  ])oint<'rs  independent  of  the  interning  set  \'icc. 
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The  solution  adopted  in  MultiSchenie  is  to  add  a  new  data  type,  the 
weak-cons.  to  Selu'ine.  A  weak-cons  is  like  an  ordinary  (‘(jns  cell  excej)t 
that  the  car  of  a  weak-cons  is  replaced  by  ’  ()  when  the  garbage  collector 
discovf'i's  that  its  former  contents  are  no  longer  rf'ferenced  elsewhere  in 
the  system.  From  the  point  of  view  of  a  })rogrammer.  the  creation  and 
us<'  of  a  weak-cons  is  similar  to  that  of  an  ordinary  cons  cell,  excejit  that 
diffeii'iit  selectors,  constructors,  and  mutators  are  used.  Weak-cons  cells 
are  very  efficient  to  create  and  reference  (unlike  the  interned  uni<iue  IDs 
they  replace),  although  they  do  impose  some  overhead  at  garbage  collection 
time. 

Tlie  unusual  handling  of  the  weak-cons  is  divided  into  two  parts,  both 
occurring  during  the  garbage  collection  cj'cle.  First,  the  actual  weak-cons 
data  object  is  handled  specially  by  the  GC  primitive.  An  ordinary  cons 
cell  would  be  copied  from  old  space  to  new  space,  and  then  the  contents 
of  its  car  and  edr  would  ultimately  be  scanned  and  copied  as  well,  as 
shown  in  Figure  2.1.  Instead,  the  contents  of  a  weak  cons  are  deliberately 
miscopied  as  illustrated  in  Figure  2.2;  the  edr  is  copied  as  usual,  but  the 
car  is  copied  and  marked  as  a  non-pointer.  Thus,  when  the  car  and  edr 
are  encountered  later  in  the  garbage  collection  cycle  the  car  is  not  copied 
but  remaijis  with  the  address  of  the  original  contents  in  old  space.  During 
this  (mis)copy  operation  the  original  weak-cons  object  in  old  space  is  also 
modified.  Its  car  is  replaced  with  a  ‘’broken  heart”  (forwarding  pointer) 
(as  is  the  car  of  an  ordinary  cons  cell)  but  its  edr  is  also  changed.  The  edr 
is  modified  to  contain  the  type  code  of  the  old  car  (this  was  lost  when  the 
copy  was  marked  as  a  non-pointer)  and  a  pointer  to  the  previous  weak-cons 
cell  encountered  by  the  garbage  collector. 

The  net  result  of  this  skullduggery,  pictured  in  Figure  2.2,  is  that  all  of 
the  information  nece.ssary  to  reconstruct  tlie  contents  of  the  car  of  a  weak- 
cons  is  available,  and  yet  it  has  not  actually  been  updated.  Furthermore, 
there  is  a  chain  (in  old  space)  of  all  of  the  weak  con.ses  encountered  during 
the  garbage  collection.  An  important  aspect  of  this  design  (in  fact,  a  driving 
concern  )  is  that  no  additional  space  is  required  for  this  handling  of  the  weak- 
cons  cf'lls.  The  actual  copying  of  a  weak-cons  is  somewhat  slower  than  the 
c(jpying  of  a  cons  cell,  but  the  difference  is  very  .small  indeed. 

The  second  i)art  of  the  work  is  performed  by  a  new  gc  demon.  The 
demon  walks  down  this  chain  and  updates  the  new  space  copies  with  either 
a  pointer  to  the  relcjcated  contents  of  the  car  or  a  '().  This  decision 
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(Before  Scanning  the  Cell^) 

Old  Space  New  Space 

Addr  Contents  11  Addr  Contents 


(After  Scanning  the  Cell^’^) 


Notes: 

1.  Assume  the  variable  PreviousWeakCons  contains  the  value  previous. 

2.  The  variable  PreviousWeakCons  will  now  contain  the  address  a. 

This  Itjpe  style  indicates  changes  from  the  handling  of  a  normal  cons  cell. 
4.  This  cell  will  contain  ’  ()  if  the  object  no  longer  exists. 


Figure  2.2:  Copying  a  weak-cons 
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is  based  on  examining,  in  old  space,  the  original  contents  of  the  car  of 
the  weak-cons.  If  it  contains  a  broken  heart  then  the  object  was  copied 
during  the  garbage  collection  and  hence  is  still  in  the  system.  If  not,  then 
the  garbage  collector  did  not  encounter  the  object  and  hence  it  has  been 
removed  from  the  system.  This  new  GC  demon  requires  time  proportional 
to  the  number  of  weak-conses  in  the  system  (but  no  additional  space). 
Hence  the  garbage  collection  is  somewhat  slower,  although  the  number  of 
weak-conses  is  small  enough  that  this  change  is  also  negligible. 


Notice  that  it  is  critical  that  this  operation  be  performed  after  the 
garbage  collector  has  been  run  to  completion  so  that  all  objects  remaining 
in  the  system  will  have  been  moved  from  old  space  to  new  space.  It  also 
requires  access  to  the  contents  of  old  space  and  hence  cannot  be  (safely) 
written  in  Scheme.  Finally,  it  is  important  that  none  of  the  GC  demons 
executed  before  this  one  in  any  way  reference  a  weak-cons,  since  all  of  the 
weak-conses  in  the  system  are  in  an  inconsistent  state  until  this  demon  has 
been  run. 


These  simple  changes  allow  MultiScheme  to  efficiently  implement  weak 
pointers.  The  population  abstraction  is  easily  implemented  using  weak- 
cons,  as  are  most  of  the  other  existing  uses  of  object -hash  and  object- 
unhash.  But  they  have  not  eliminated  the  need  for  the  hash  demon,  since 
there  is  no  support  for  the  interning  services  of  that  demon.  However,  once 
the  system  contains  these  weak-cons  objects,  the  hash  tables  (previously 
special  objects  hidden  from  the  view  of  the  garbage  collector)  can  be  re¬ 
placed  by  ordinary  tables  constructed  from  regular  Scheme  data  structures 
and  using  a  weak-cons  to  hold  the  keys.  The  tables  still  need  to  be  re¬ 
built  after  each  garbage  collection  since  the  hash  is  based  on  the  address  of 
the  objects  in  memory,  but  they  are  no  longer  deeply  intertwined  with  the 
workings  of  the  garbage  collection  algorithm.  Furthermore,  by  separating 
the  two  services  the  size  of  the  hash  tables  is  significantly  reduced,  since 
they  need  only  contain  objects  passed  to  the  actual  interning  service.  As  a 
lesvdt,  the  time  spent  during  each  garbage  collection  for  the  maintenance 
of  the  hash  tables  is  reduced  and  the  overall  amount  of  time  spent  during 
garbage  collection  is  reduced. 
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2.2.3  Object  Finalization 

The  file  demon  siipplies  a  s}'steni  service  of  a  very  specific  nature.  It  detects 
the  disappearance  of  certain  objects  (files)  in  the  system  and  exeentt's  a 
specific  jnocedure  for  each  object  (it  closes  the  file).  But  cU'arly  it  would 
be  nice  to  generalize  this  to  allow  for  arbitrary  objects  to  be  "finalized" 
when  they  are  no  longer  needed  by  the  system.  This  would  b('  useful, 
for  example,  as  the  basis  for  a  resource  system  designed  to  allow  tasks  to 
allocate'  s('lect('d  resources  while  assuring  that  they  will  be  rt'leased  t've'ii  if 
the  task  "dies"  in  some  way. 

Such  a  resovirce  system  is  an  object-oriented  version  of  tlu*  tdeaii-up  code' 
preevided  by  a  language  like  Mesa[41],  and  is  modeleel  on  the  finishing  ceiele 
eif  Praxis[oS|.  In  Me-sa.  e-le-an-up  code'  is  attacheel  as  an  exe'eption  handler 
te)  a  specifie-  profjrain  fragment  te)  allejw  actiems  to  be  take'ii  if  that  fragine'ut 
e'ue'ounte'rs  an  e'lreer.  These'  actions  can  include,  for  example,  releasing  loe-ks 
ejr  ele'-alleicating  objerts  cre-ated  by  the  program.  This  bc'havior  can  be  erb- 
taine'el  using  existing  nu'chanisms  in  Scheme.  The  new  me'chanism.  heewe-ver. 
attaches  the'  e'le'an-up  code  to  specific  objects  in  the  system.  It  is  in  some' 
se'iise'  the'  logictil  inverse  eef  initialization  code  run  when  an  object  is  created, 
teeing  run  inste'ael  whe'ii  the  object  is  removed  from  the  system.  The  notion 
of  some  jearticular  area,  of  program  responsible  for  clean-up  is  replaced  by 
the  notion  of  a  self-cleaning  object. 

Since  the  critical  issue  here  is  again  the  “disappearance”  of  an  object 
from  the  systt'in,  it  is  clearly  related  to  the  garbage  collector.  And  again,  a 
simple  extension  to  tin  gtirltage  collector  provides  a  good  base  for  this  gen¬ 
eralization.  MtiltiScheme  tillows  the  system  to  provide  one  object,  called  the 
finalization  object,  to  l>e  tised  as  a  root  for  the  primitive  GC  after  all  of  the 
other  roots  have  been  scanuc'd.  The  primitive  GC  reports  back  the  location 
into  which  this  rofit  w;is  copied  .m)  that  objects  stirviving  the  garbage  collec¬ 
tion  normally  c;m  la'  distinguished  from  those  surviving  because  they  were 
reacluible  only  from  this  .  jx'cial  root.  This  techniejue,  of  course,  depends 
uj)on  the  f.ict  that  sitace  for  copying  objects  from  old  space  is  allocated 
se(juenti;dly  in  new  sjaice  during  the  garbage  collection.  While  this  ha.s  not 
actually  Ix'en  iniplemeut<'d  on  the  parallel  processor,  the  extension  to  that 
implementation  is  strtughtforward.  On  toj)  of  this  very  low-level  mechanism 
it  is  easy  to  provid('  an  I'xtt'usibh'  meclumism  for  ol)ject  fimdization. 

.MiiltiScht'ine  int('rleav('s  the  supj)ort  for  finalization  and  weak-conses  in 
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a  well-defined  order.  It  first  performs  all  of  the  normal  garbage  rollection 
cycle  except  that  the  car  of  a  weak-cons  is  not  copied.  Next  it  coj^ies  the 
finalization  object  and  notes  where  in  memory  this  copying  process  began. 
Finally  it  runs  the  user  supplied  demons,  one  of  wdiich  updates  the  car 
of  the  vveak-conses  to  reflect  the  existence  of  objects  retained  either  in  the 
ordinary  process  or  by  the  finalization  object.  As  a  result,  data  structures 
constructed  using  a  weak-cons  do  allow  the  objects  to  which  the  car  refers 
to  be  finalized.  This  has  two  ramifications.  First,  a  resource  system  can  be 
constructed  using  finalization  code  as  a  means  of  granting  serialized  access 
to  a  resource.  The  data  structures  used  by  the  system  must  be  built  using  a 
weak-cons  to  allow  the  finalization  code  to  run  even  though  the  system  can 
continue  to  reference  the  objects  (via  the  weak-conses).  Second,  it  is  7iot 
correct  for  finalization  code  to  assume  that  there  are  no  references  to  the 
object  remaining  in  the  system.  Rather,  the  correct  assumption  is  that  the 
only  remaining  references  are  by  way  of  the  weak-cons  mechanism.  This 
interaction  between  weak-cons  and  finalization  is  useful,  but  it  does  recpiire 
the  exercise  of  care  when  using  both  mechanisms  in  the  same  system. 

This  is  an  easy  and  efficient  way  to  implement  finalization  code.  It  in¬ 
tegrates  conveniently  into  the  system,  allowing  certain  forms  of  consistency 
recpiirements  to  be  guaranteed.  There  are,  however,  some  drawbacks  that 
luait  its  usefulness.  They  derive  from  two  different  aspects  of  garbage  col¬ 
lection.  First,  there  is  no  way  to  predict  the  amount  of  time  before  a  freed 
resour^'e  will  be  cleaned  up.  As  a  result,  at  least  for  resources  in  frecpient 
demand,  this  mechanism  must  be  looked  on  as  a  form  of  fail-safe  device. 
It  should  be  used  only  in  cases  where  it  is  essential  to  the  system  that  a 
final  operation  Ik-  performed  to  maintain  an  important  consistency  recjuire- 
ineiit  even  when  the  ordinary  mechanisms  for  releasing  a  resource  may  not 
b<'  invoked  by  the  user's  program.  Alternatively  if  a  task  discovers  that  a 
resource  it  requires  is  not  available  it  may  choose  to  w'ait  for  an  interval 
sufficient  for  the  resource  to  be  released  and  then  force  a  garbage  collection 
if  the  resource  is  still  not  available.  Thus,  the  finalization  code  approach 
help>  solve  the  problem  of  a  "disappearing  resource,”  but  it  docs  not  truly 
-'Upj)ort  automatic  releasing  of  a  critical  resource. 

Tie-  second  difficulty  is  somewdiat  more  subtle.  An  object  is  finalized 
only  when  all  methods  of  locating  it  (except  through  w’eak-cons  cells)  have 
111  ell  removed  from  the  system.  Tims,  if  an  object  is  placed  in  a  data  struc- 
niie  by  some  task  that  "owns"  the  ol>ject.  the  object  will  not  be  finalized 
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until  the  (lata  stiiu'tnic  licronies  inaccessible  and  the  task  loses  any  means 
to  reference  the  obji'ct  directly.  This  is  a  t\vo-edj;;ed  sword.  For  some  s}'s- 
tem  oi^ani/atious  it  is  perh'ctly  reasonable  to  assume  that  an  object  is 
in  ust'  as  lonj;  as  it  can  be  found  by  any  method  whatsoever.  For  other 
'ystems.  howercr,  it  may  b<'  better  to  view  an  obj('ct  as  in  use  only  as 
loiiit;  as  it  can  be  accesst'd  in  particidar  waj's  rep;ardless  of  other  ways 
the  i;aibaye  collector  miy;lit  be  able  to  come  across  it.  Whih*  MultiScheim' 
does  not  support  any  general  mechanism  to  help  this  latter  case,  at  least 
one  common  case  is  easily  accommodated.  If  the  access  path  of  inter('st  is 
a i war's  \-ia  a  known  set  of  objects  (for  example,  a  resourct-  might  be  con- 
saleiH'il  inacccssibh'  when  the  task  that  allocatt'd  it  disapjx'ars ).  then  these 
objects  lather  than  the  resource  can  be  the  subject  of  finalization.  Thus, 
a  task  might  contain  a  list  of  all  the  resources  it  has  allocated  and  not 
released.  The  finalization  cod<'  for  each  task  would  then  examine  the  list 
of  rt'sources  accpured  by  the  about-to-disappear  task  and  assure  that  they 
are  all  reltuvsed. 


2.3  Example  1:  Logic  Variables  in  Lisp 

With  the  recent  inter<'st  in  "logic  programming  languages,”  a  good  deal  of 
atft'ntitin  has  been  paid  to  techniques  for  merging  logic  programming  into 
Lisp  40;.  There  arc'  thrt'e  important  notions  inherent  in  tlutse  langmiges 
trom  the  imiih'iiK'iitor's  jroint  of  view:  unification,  backtracking,  and  em¬ 
bedding  variables  in  dtita  structures.  Unification  algorithms  can  be  easily 
written  in  Lisp  (see,  for  example,  the  one  in  [7],  Section  4.5).  Similarly, 
complieated  control  structures  involving  backtracking  (iroth  temporal  and 
deix'iidency-directed)  can  be  expressed  in  Scheme  using  a  combination  of 
first-class  cont iimations  and  procetlures.  Thus,  they  pose  no  difficidties 
when  attemi)ting  to  embed  a  logic  hmguage  within  Lisp.  The  logic  variable, 
howevc’r.  can  be  emix'dded  in  several  data  structures  iir  such  a  way  that 
lu'oviding  a  value  to  the  variable  in  any  structure  causes  that  value  to  ap- 
[>'  ar  in  all  of  fh<'  data  structures.  This  is  where  a  clean  embt'dding  in  Lisp 
b(  c  oiner  ditficult;  a  good  embedding  should  permit  hjgic  variables  to  be 
pait  ot  arbitrary  dat.a  objects  and  when  the  variable  receives  a  value  all  of 
the  data  object.^  shotdd  conttun  this  value,  for  use  in  whatever  manner  that 
vtdue  woidfl  norundly  be  manipulated.  It  is  precisely  here  that  the  addition 
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of  plact'holders  to  Scheme  plays  a  role.  By  representing  logic  variables  us¬ 
ing  placeliolclers.  provided  we  do  not  in  some  way  touch  them  before  their 
\alues  are  comptited,  we  need  never  concern  ourselves  with  whether  the 
data  stnictuK'  was  made  u.siug  normal  Scheme  techniciues  or  through  the 
use  of  unification. 

The  remainder  of  this  s('ction  ch'monstrates  an  implementation  of  logic 
variables  using  the  mechanisms  already  discussed.  The  focus  here  is  on  the 
form  of  sharing  i)rovided  by  logic  variables.  Thus,  the  implementation  dotes 
not  attempt  to  addrt'ss  issues  such  as  backtracking^  or  the  deeper  cpicstions 
of  logic  raised  by  the  particular  unification  algorithm  shown  here. 

Let  us  represent  a  logic  variable  as  a  placeholder,  initially  mutably  de- 
ttuinined  to  a  detectable  value.  In  this  case,  it  is  the  name  of  the  variable 
(for  us  to  look  at)  and  a  special  tag.  The  placeholder  is  mutably  determined 
so  that  when  it  successfully  unifies  with  an  expression  we  can  change  the 
value. 

(define  variable-tag  ’(LOGIC  VARIABLE)) 

(define  (make-variable  name) 

(let  ((result  (make-placeholder))) 

(mutably-determine!  result  (cons  name  variable-tag)) 
result) ) 

(define  (variable?  object) 

(and  (pair?  object) 

(eq?  (edr  object)  variable-tag))) 

If  we  unify  a  logic  variable  with  an  expression,  the  variable  must  hence- 
f(jrth  take  on  the  value  of  the  expression.  Since  we  are  not  concerned  with 
liacktracking  we  can  use  the  standard  determine!  operation  to  accomplish 
this: 

(define  (unif y-variable  variable  expression) 

(determine!  variable  expression)) 


■’  lo  provilif  a  version  of  the  unifier  that  supports  backtracking  is  not  a  difhcult  task, 
but  riM|uir('.s  more  code  tlian  sliown  here.  The  major  differences  are  to  pass  explicit  con¬ 
tinuations  for  use  wlien  a  unification  fails,  and  to  use  mutable  (rather  than  immutable) 
determine!  to  provide  the  values  of  variables  that  are  successfully  unified. 
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T>)  unify  twx)  exprussious  we  must  haiulle  a  number  of  cases.  If  the 
expressions  are  identical,  then  they  uuif\’  to  themseh'es.  If  either  is  a  vari¬ 
able.  then  we  can  use  unif y-variable  to  combine  it  with  the  otlu'r.  If 
lioth  expressions  are  composed  of  sub-expressions  we  recursively  unify  the 
sii!  lexpii'ssions.  This  ]>articular  unification  procedure  will  return  #t  if  the 
rwd  exjuessiiuis  cau  bt'  successfully  unified,  or  #f  if  they  cannot.  .After 
■-la  ce-sful  unification,  logic  varialdes  in  either  expression  have  acquired  the 
\aiues  of  the  corresponding  coni]H)m'nts  of  the  other  expression.  This  is 
easil}'  impleiiK'nted: 

(define  (unify  expl  exp2) 

(cond 

((eq?  expl  exp2)  #t) 

((variable?  expl)  (unif y-variable  expl  exp2)  #t) 

((variable?  exp2)  (unify-variable  exp2  expl)  #t) 

((and  (pair?  expl) 

(pair?  exp2) 

(unify  (car  expl)  (car  exp2))) 

(unify  (edr  expl)  (edr  exp2))) 

(else  #f))) 

.Vofice  that  this  unifier  i.s  simpler  than  a  more  traditional  unifier  (such 
as  the  one  in  [-!)  written  in  Lisp.  Because  a  determined  placeholder  is 
usi'd  to  represent  a  variable  after  it  has  been  unified  it  is  not  necessary 
to  consider  this  case  si)ecially.  A  determined  placeholder  is  the  value  it  is 
unified  with,  and  thus  once  a  variable  is  unified  it  no  longer  matches  the 
variable?  chauses. 

.As  mentioned  at  the  out, set,  this  program  docs  not  address  the  issues 
related  to  unification  and  backtracking.  It  lacks  two  features  important  in 
a  logic  progrannning  system:  an  occur  check,  and  backtracking.  The  occur 
check  is  ea.'ily  addecl,  but  at  a  large  performance  penalty  (the  occur  check 
is  ouiittcil  in  Prolog  for  this  same  reason).  Furthermore,  if  thi.s  unifier  fails 
to  unify  two  expressions  it  may  still  have  determined  the  values  of  some  of 
die  variables  in  the  expressions.  This  problem  is  easily  solved  in  a  version 
written  to  include  ;i  l>acktracking  contnd  structure:  unification  of  variables 
could  be  done  with  a  mutalrle  detruinine  and  backing  out  br-yond  the  point 
wiu’re  it  was  unified  wotiUl  restore  the  jtrevious  value. 

Using  the  progr.am  siiown  here,  we  can  create  some  Sclu'me  wuittbh's 
wliose  \alue.',  will  be  cieated  Ijy  the  jirocess  of  unification: 
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;  Create  some  "logic  variables" 

(define  ?a  (make-variable  ’a)) 

(define  ?b  (make-variable  ’b)) 

(define  ?c  (make-variable  ’c)) 

;  Some  simple  expressions  including  the  variables  for  unification 
(define  EXFl  (list  ’A  ?a  ’B  ?b)) 

(define  EXP2  (list  ’A  10  ’B  ?c)) 

;  And  a  "Schema"  expression  not  for  unification 
(define  SCHEME-EXPR  (list  ?a  ?b  ?c)) 

;  Try  it  out . . . 

SCHEME-EXPR  ^  (# [FUTURE  16]  # [FUTURE  84]  # [FUTURE  52]) 

;  None  of  the  variables  has  an  (immutable)  value  and  all  are  different 

=>  (unify  EXPl  EXP2)  #t 

=>  SCHEME-EXPR  'v*  (10  #  [FUTURE  64]  #  [FUTURE  64]) 

;  ?A  has  a  value,  but  ?B  and  ?C  do  not.  ?B  and  ?C  are  identical. 

=>  (unify  ?b  3)  #t 
=>  SCHEME-EXPR  ^  (10  3  3) 

=>  (apply  +  SCHEME-EXPR)  16 
=>  expl  (A  10  B  3) 

=>  exp2  (A  10  B  3) 

By  embedding  “logic  variables”  in  the  expression  SCHEME-EXPR,  this 
expression  changes  as  the  variables  receive  values.  Most  importantly,  as 
shown  in  the  expression  (apply  +  SCHEME-EXPR),  it  is  not  necessary  to 
know  which  elements  of  the  data  structure  received  their  value  by  unifica¬ 
tion  and  which  received  them  through  other  mechanisms. 


2.4  Example  2:  Normal  Order  Evaluation 

One  of  the  fundamental  results  of  the  A-calculus  is  the  Church-Rosser  Theo¬ 
rem  -  a  proof  that  (in  a  system  without  side-effects)  any  order  of  argviment 
evaluation  leads  to  the  same  answer  provided  the  process  terminates.  The 
tlieorem  also  shows  that  one  particular  order  of  application,  called  normal 
order,  will  terminate  if  any  order  will  terminate. 


I.- M  V  V K"  H.'  ■■■  ■■■  » ■  »"  V ■.■  »■  f  T*  ■■" »■' 

. 


_  »  •  »  »  ^ 


« 


4S  CHAPTER  2.  EXTENDED  SEQUENTIAL  SCHEME 

The  esst'uee  of  iionual  order  evaluation  is  “substitute  first”  on  function 
call  uiievaluated  arguments  are  substituted  into  the  body  of  a  function 
before  the  body  of  the  function  is  evaluated.  Expressions  tend  to  expand 
during  a  substitution  pha.se  and  then  contract  in  a  reduction  phase.  While 
there  are  few  theoretic.'d  results  to  support  a  general  assertion,  it  appears 
that  this  exptinsion  during  the  substitution  phase  yields  an  unaccepttible 
.^piLci'  ref[tiirement  in  actual  implementations.  By  contrast.  Scheme  em¬ 
ploys  applicative  order,  as  do  most  other  programming  languages,  in  which 
the  tirguments  ;ire  evaluated  prior  to  evaluating  the  body  of  the  functicjii. 
.Attempts  to  reduce  the  space  complexity  of  pure  normal  order  systems 
have  uuide  very  little  headway.  For  example,  even  a  simple  iterative  facto- 
ri;d  program  that  executes  in  constant  space  in  (applicative  order)  ScheuK’ 
requires  a  linear  amount  of  storage  in  (normal  order)  Miranda.  This  is 
currently  a  major  research  area  for  the  functional  languages  community. 

Uufortvmately.  the  use  of  applicative  order  for  function  call  leads  to 
infinite  loops  or  errors  in  some  programs  that  would  terminate  correctly 
under  normal  order.  A  classic  example  of  this  problem  is  the  if  special 
form  of  Scheme.  Consider  a  modified  version  of  Scheme  where  true  and 
false  are  implemented  as  in  Church’s  work,  and  predicates  (including  =) 
n't  urn  one  of  these  new  values: 

(define  true  (lambda  (X  Y)  X))  ;  Replaces  IT 

(define  false  (lambda  (X  Y)  Y))  ;  Replaces  #F 

In  such  a  system  we  might  attempt  to  define  our  own  function,  following 
Church,  expecting  it  to  work  like  the  standard  if  special  form: 

(define  (our-if  predicate  consequent  alternative) 

(predicate  consequent  alternative)) 

Finally,  we  could  try  it  out  as  follows: 

(define  x  0) 

(our-if  (=  X  0)  0  (/  10  x)) 
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In  Scheme,  or  any  other  applicative  order  system,  the  final  line  will  n’sult 
r.i  a  l  uiifinii-  t  rror.  The  three  arguments  to  our-if  will  all  be  evaluated 
lufiu-f  the  body  f)f  our-if  is  examined,  and  the  third  argvmient  (  (/  10  x)  ) 
will  gi\e  a  di\  ide  l)y  zero  error.  By  contrast,  a  normal  order  system  would 
-".b'f ire.tc  rlic  expressions  themselves  into  the  body  of  the  function.  Thus 
'hi-  fii.al  liiii'  would  be  transformed  in  successive  steps  as  follows: 
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(our-if  (=  X  0)  0  (/  10  x)) 

((=  X  0)  0  (/  10  x)) 

((lambda  (X  Y)  X)  0  (/  10  x)) 

0 

Thus,  the  programming  community  seems  to  be  faced  with  an  unfortu¬ 
nate  trade-off.  Normal  order  systems  are  convenient  because  they  get  stuck 
in  computational  blind  alleys  only  when  there  is  no  other  choice.  Applica¬ 
tive  order  systems  are  space  efficient  because  they  produce  data  objects 
at  an  earlier  stage  of  the  computation.  A  variety  of  avenues  have  been 
explored  for  combining  the  best  parts  of  both  kinds  of  systems: 

•  Strictness  Analysis,  a  static  technique  for  discovering  which  arguments 
to  a  procedure  can  be  passed  by  value  (i.e.  using  applicative  order) 
without  running  into  the  attendant  difficulties. 

•  Syntactically  marking  entire  procedures  as  “by  value”  or  “by  name.” 
giving  programmer  control  over  the  trade-off  on  a  procedure  by  ])ro- 
cedure  basis. 

•  Marking  individual  procedure  parameters  as  “by  value”  or  “by  name." 

•  Allowing  individual  calls  to  procedures  to  pass  delayed  arguments. 
This  is  the  approach  permitted  in  Scheme;  unfortunately,  the  proce¬ 
dure  must  be  written  so  that  it  anticipates  which  arguments  are  (or 
may  be)  delayed. 

Placeholders  provide  a  time-efficient  vehicle  for  solving  the  difficulty  men¬ 
tioned  in  this  last  approach.  With  the  addition  of  a  placeholder  object  the 
iK'cd  to  explicitly  force  an  argument  is  replaced  with  the  automatic  touch 
supplied  for  placeholders.  Thus  procedures  can  be  called  with  a  “normal- 
order"  parameter  by  merely  specifying  a  placeholder  for  that  parameter. 
The  code  for  the  procedure  is  unchanged,  and  can  thus  handle  any  set  of 
its  parameters  being  passed  using  either  method. 

In  order  to  make  this  use  of  placeliolders  convenient,  MultiScheme  j:)]  o- 
\'ides  a  standard  scheduling  policy  known  as  the  delay-policy  (see  Sec¬ 
tion  4.3.1 ).  This  policy  cainses  the  placeholder  to  contain  a  procedure  (his- 
toiically  known  as  a  thunk)  that  will  be  called  when  the  value  of  the  pa- 
rana'ter  is  required.  The  scheduler  resjjonds  to  a  touch  of  tins  phicehoUh-r 
by  calling  the  tlmiik  and  determining  the  value  of  the  placeholder.  Tims, 
we  could  rt'write  our  earlier  trial  jirograin  as: 
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(our-if  (=  X  0)  0  (future  (/  10  x)  delay-policy)) 

it  will  now  work  witlujut  any  otlier  c!ian”;('s.  protluciiig  the  (l(‘sir('(i 
0  n'Milt. 

Tliu>,  in  much  flu'  ^aIue  way  that  the  fntui'f'  niechaiiisiii  allows  manual 
annotation  of  nartilh'lism  it  also  allows  manual  annottuion  of  iionmil  order 
’  an  a:;irr<-r  pas'iny'.  Ona  j)art  ieularly  interesting  us('  of  this  ability  is  in 
rel.at'o;;  u>  the  letrec  eonstruet  used  to  define  a  set  t)f  mutually  r('rursi\'(' 
tdciitihers.  Seheme  provides  a  special  form  for  this  pnirpose  whose  syntax 
i-  the  'aiue  as  that  of  the  let  specitil  form.  Thf-  standard  Scluane  reference 
uniuiiid  49_,  however,  sttites; 

One  restriction  on  letrec  is  very  important:  it  must  he 
possilile  to  evahuite  each  [initial  value  expression]  without  re¬ 
ferring  to  th(>  value  of  any  [of  the  introdticed  variables].  If  this 
restriction  is  violatt'd,  then  the  effect  i.s  undefined  .  .  .  The  re- 
'tiiction  is  uecessttry  because  Scheme  passt's  arguimnits  by  vtilue 
rarlicr  rhtm  by  ntune.  In  the  most  common  uses  of  letrec.  ;dl 
the  [initial  value  expressions]  are  lambda  expressions  and  the 
restriction  is  stitisfied  autcjinatically. 

Ihe.s,  the  letrec  sjtecittl  form  in  Scheme  is  retdly  not  intended  for  use  in 
c;  e:iriug  recursive  dat;t  strtictures  or  any  other  ehdtorate  mutmdly  recursive 
d' iiui''.o:i.  By  using  the  phiceholder  implementation  of  dehiytal  obj('cts. 
ho’.\e\-i  r.  this  la'striction  <';in  b(‘  elimimitcal.  In  hn't  :i  simple  imicro  c:ui  la' 
e.-^eii  to  trtmslafe  ti  letrec  exjtres.-.'icjn  into  :tn  ecpiivaleiit  let  expression. 
The  :i<  w  expres.-iou  will  coim-rge  to  ;i  value  whenever  tin'  origimd  letrec 
will  coii\-erge.  Consider  the  circuhir  list  crt'titevl  by 

(letrec  ((ones  (cons  1  ones))) 
ones)  ;  body 

I  li  aii<  tii  tu;  i-ijn:  in  Sch<'me.  since  th<'  \';irial)le  ones  is  introduced 
letrec  l'':;;i  .•Ciidi  a.lso  U'-cd  m  the  inititil  \'idue  form.  This  can  be 
:  ii  e; hy  fiaiu.afi  d.  howe'i'er,  into  the  following  e.xpressiou: 

I  I  ■!'  '.mC.  il  li.-|.  i>  iinl  ..iiilii'i'  lit  to  pr>.v  iil'  full  'I'.'ill  liy  iciiiii'’  ].ar:uii'- 

' '  :  '  1 '  1  '■  y  i ; i .■  v,  nil  ;i  1 1  .u  t  lir  1 \  i  ,1  tin-  j  .ri  h  i-.  1  lu'i-  n  i  i mi t  a i  .■  1 1 ir  |.  .rii lal  )  ..irai i ii  u  r 

e.  :  '  r.  si.|.-.il..i  !.•  I :  .iii>n  ill  I 'i  1  Isi'k  to  ill'-  argiiiiK'iit ,  1  In  iii.rhaiieiii  i>  iiiMr'- 

;i  :  '  ali  l.y  ii''.!  •  iiiiiiii)i/'.i| )  <  p|ii  iini/al  ii  mi  i  f  ir  M'liial  orilrr  I'v  aliiat  n  mi  . 
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ji 

(let  ((ones)) 

;  Create  variable,  no  value 

(set!  ones  (future 

(cons  1  ones)  delay-policy)) 

;  "Fucurized  delay"  of 

t'; 

;  initial  value  expression 

ones) 

;  body 

a-: 


fc-: 


>y 
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V. 


This  correctly  implements  the  version  of  the  original  letrec  expression. 
A  similar  syntactic  transformation  can  be  used  for  any  letrec  expre'ssion, 
effectively  removing  the  language  restriction  quoted  earlier.  Notice  that 
this  same  transformation  would  not  work  using  the  delay  special  form  of 
Scheme;  my  method  depends  on  the  automatic  touch  provided  for  place¬ 
holders  (but  not  for  ordinary  delayed  objects). 


2.5  Example  3:  Stream  Processing 


One  very  interesting  technique  for  structuring  (primarily  functional)  sys¬ 
tems,  called  stream  processing,  involves  the  use  of  a  data  structure  (a  stream) 
similar  to  a  list  but  with  the  edr  of  each  cell  containing  a  delayed  object 
iiistcad  of  the  actual  object.  Thus  as  one  recursively  walks  down  a  stream 
the  first  item  (its  car)  is  always  available  but  the  remaining  items  must  be 
forced  in  order  to  access  their  value.  Since  only  the  items  actually  needed 
during  a  computation  are  forced  during  that  computation,  a  stream  can 
represent  an  infinite  set  of  values  while  occupying  a  finite  amount  of  space. 
If  the  stream  “remembers”  the  values  of  items  already  referenced  so  that 
they  needn’t  be  recomputed  (called  memoization),  then  the  data  structure  is 
efficient  in  both  time  and  space.  Streams  are  easily  implemented  in  Scheme, 
and  lead  to  a  very  convenient  abstraction. 

An  excellent  introduction  to  stream  processing  in  Scheme  is  found  in 
Abelson  and  Sussman[7],  Section  3.4.  They  demonstrate  how  streams  are 
used  in  a  variety  of  examples.  Two  of  these  examples,  in  particular,  high¬ 
light  problems  with  the  stream  implementation  as  normally  supported  in 
Scheme:  the  need  for  uniformly  delaying  arguments  to  procedures  (in  the 
integrate  proc(xlure),  and  the  difficulty  of  producing  a  fair  merge  oper¬ 
ation.  By  using  placeholders  rather  than  the  standard  implementation  of 
flelayed  objects  MulfiScheme  provides  solutions  to  both  of  these'  ])robl('ms. 
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2.5.1  Uniformly  Delayed  Arguments 

TIk'  first  (liffictilty  tirises  out  of  the  iieetl  for  imitually  ri'i-ui"'i\i'  srri  ;iiii'  ii; 
oi  lier  to  solv('  cc'i'tain  simple  ])roblems.  The  example  expKnei!  in  A1 'e!-,  ,i. 
and  Sussinaii  is  a  simjde  (numerical)  iiitef;;rator  u-'ed  to  ^ol\e  diif.  i ei.t nd 
I'qnations.  Tlu' development  of  that  example  is  re])eated  Imie  i  in  conden-i  i! 
form)  to  illustrate  the  general  problem. 

It  is  etisy  enough  to  write  a  procedur*'  tluit  receire>  input  a  -freain 
of  incoming  sample  values  (integrand),  wie  value  of  the  integral  at  tune 
/  =  0  (initial-value),  and  the  time  interval  between  >(aniiiles  idti  and 
luoduces  a  stream  of  the  integrated  values; 

(define  (integral  integrand  initial-value  dt) 

(define  int 

(cons-stream  initial-value 

(add-streams  (scale-stream  dt  integrand) 
int))) 

int) 

Rt'call  that  the  second  argument  to  cons-stream  is  delayed  rather  than 
evnluatt’d,  so  the  fact  that  it  references  the  value  of  int  as  part  of  the 
definition  of  int  does  not  lead  to  a  problem.  There  is  a  built-in  vmit  of 
delay  in  this  branch  of  the  stream. 

We  might  try  to  use  this  procedure  as  part  of  a  program  to  solve  a  first- 
order  differi'utial  eipratiou.  Given  a  procedure  that  calculates  the  value 
of  fiy).  w('  want  to  find  th<>  stream  of  values  y(f)  such  that  ^  =  fil/)- 
This  would  lie  straightforward  if  Scheme  supported  mtitually  recursiv(>  data 
structure  definitions  (for  (>xami)le,  if  Scheme  were  a  normal  order  language). 
In  that  case  we  could  write: 

(define  (solve  f  initial-y-value  dt) 

(define  y  (integral  dy  initial-y-value  dt)) 

(define  dy  (map-stream  f  y)) 

y) 

Unfortunateh'.  in  order  to  ctdculate  the  value  of  y  (a  stream)  we  must 
supply  the  value  of  dy  (:ilso  a  stream)  to  the  integral  procedure.  But  m 
Ollier  to  calculate  dy  we  need  to  know  the  v;due  of  y.  One  wtty  out  of 
thi-  dilemma  i-  to  explicitly  introduce  a  delay  operation  to  “cut  the  loop 
b\'  liuinging  fill'  infegrtim!  argtiment  from  ctill  by  value  to  ctill  by  nam<'. 


2.5.  EXAMPLES:  STREAM  PROCESSING 


53 


In  MIT  Scheme,  however,  this  requires  that  the  body  of  integral  have  a 
corrt'spoiiding  force  added.  This,  in  turn,  requires  that  all  calls  to  integral 
ha\  e  a  tlelayed  argument  for  the  integrand. 

The  problem,  then,  is  that  the  modularity  of  the  program  is  disrupted. 
Because  one  call  to  the  integral  procedure  needed  to  be  delayed,  the  external 
interface  to  the  procedure  was  altered,  and  all  calls  need  to  be  located  and 
changed  correspondingly.  This  is  the  same  problem  explored  in  Section  2.4 
appt'aring  in  a  different  guise.  It  is  not  surprising,  then,  that  the  same 
sohition  can  be  used.  In  MultiScheme  it  is  not  necessary  to  explicitly  force 
a  (U'layed  object  if  the  object  is  implemented  using  a  placeholder.  Forces 
I  touches)  are  supplied  automatically  when  the  object  is  referenced.  Thus  in 
MultiScheme  we  need  only  insert  delays  in  those  calls  arising  from  mutually 
recursive  declarations. 


2.5.2  AMB  and  Fair  Merge 

In  a  purely  functional  stream  processing  system  objects  are  modeled  by 
[)r(jviding  functions  used  to  derive  the  new  state  of  an  object  from  informa¬ 
tion  in  the  previous  state.  Furthermore,  it  is  assumed  that  these  modeling 
functions  can  be  executed  at  any  time  and  in  any  order.  This  proves  to  be 
an  extremely  elegant  formulation  for  many  self-contained  systems.  Unfor¬ 
tunately,  interactions  with  objects  external  to  the  system  may  have  a  very 
different  character.  It  is  not  possible  to  provide  a  function  describing  the 
state  of.  for  example,  a  computer  terminal  as  it  will  appear  five  minutes 
from  now.  \Ve  would  prefer,  instead,  to  model  the  terminal  as  a  device  that 
issues  ev('nts.  Thus  we  would  like  to  make  a  stream  corresponding  (say)  to 
the  secpienco  of  characters  typed  on  a  particular  terminal  line,  along  with  a 
tag  specifying  which  terminal  they  came  from  (so  we  can  support  multiple 
terminals).  This  is  simple  enough  to  implement  in  Scheme,  although  there 
is  at  least  one  oddity  here.  What  is  the  first  item  in  this  stream  (recall  that 
the  first  item  in  a  stream  is  instantly  available  and  only  subsequent  items 
may  require  a  delay  while  they  are  computed)? 

(define  (make-terminal-stream  terminal) 

(define  (loop) 

(cons-stream 

(cons  terminal  (read  terminal))  ;  Tag  +  Data 
(loop)))  ;  More  elements 

(cons-stream  ’dummy-first-element  (loop))) 


^  -  r  -  *  i 
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Xow  tlu'  stroaiii  bahaves  normally,  although  attempts  to  read  l)eyond 
flit'  firsf  iti'iii  in  tin'  stream  will  cause  the  system  to  delay  until  characters 
are  fypt'tl  on  tlie  keyboard  a  potentially  unbounded  amount  of  time. 

But  building  an  event  based  system,  a  very  productive  technique  in 
many  operating  systems,  is  not  this  simple.  Henderson’s  paper[31]  pursues 
this  same  examjtlc  to  the  next  step.  What  if  our  goal  is  to  write  a  program 
in  order  to  listen  to  two  terminals  and  process  the  key  strokes  from  them 
in  the  order  in  which  they  arrive  from  the  terminals?  This,  of  course,  is 
{irecisel}'  where  event  drir'cn  systems  are  at  their  best.  If  we  liave  more  than 
one  source  of  events  (say  we  have  two  terminals  with  a  stream  for  each)  we 
now  must  specify  in  which  order  we  will  examine  the  streams.  And  here  is 
the  fundamental  problem.  Our  program  is  no  longer  truly  free  to  api>ly  its 
state  transition  functions  in  an  order  of  its  own  choosing.  Whiche%'er  stream 
the  program  chooses  to  examine  first,  it  is  possible  that  the  corresponding 
terminal  will  not  have  any  data  available.  Yet  the  framework  described 
so  far  has  iuj  way  to  examine  a  stream  without  waiting  for  the  data  to 
iurive.  So  regardh'ss  of  which  stream  the  program  examines  first,  there  is  a 
possilhlity  that  it  will  delay  when  data  was  available  on  the  other  stream. 

In  most  event  based  systems  there  is  a  procedure  allowing  a  program  to 
say  "please  delay  me  until  any  one  of  the  following  events  occurs,  then  tell 
me  which  one  it  was’’.  This  is  precisely  the  ability  we  need  to  solve  this 
problem.  There  are  a  number  of  ways  to  formalize  and  solve  the  problem, 
all  of  them  rec[uiriug  the  addition  of  a  relation  (rather  than  a  function) 
to  the  system.  Two  such  relations  are  McCarthy’s  amb  operator[40]  and 
fair  merge.  For  my  purposes,  the  following  informal  descriptions  of  these 
relations  will  suffice; 

Amb  takes  two  inputs.  If  only  one  of  its  inputs  is  undefined,  then  the 
value  is  the  other  input.  If  both  inputs  are  undefined,  then  the  value 
is  undefined.  If  both  inputs  are  defined,  the  value  is  either  of  the  two 
(I  do  jiot  rerpiire  amb  to  be  ‘‘fair”). 

Fair  Merge  takes  two  streams  and  returns  a  stream  containing  all  of  the 
elements  from  its  inputs  in  the  same  relative  order  in  which  they 
(jcciirred  in  the  inj)uts  (hence  it  is  a  mergt').  The  output  stream  has  no 
predefined  order  of  interleaving  of  the  two  streams,  but  rather  includes 
items  from  each  stream  in  the  order  in  which  their  values  become 
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a\'ailable.  In  cases  where  values  are  available  from  both  streams  at 
til  '  same  time,  fair  merge  does  not  give  preference  to  either  stream. 

MultiScheme’s  placeholders,  along  with  specific  help  from  the  scheduler, 
provide  an  event- driven  mechanism  for  supporting  the  creation  of  the.se 
procedures.  The  basic  support  conies  from  the  disjoin  operation  of  the 
scheduler  whose  implementation  is  shown  in  Figure  4.9  on  page  lOG.  Dis¬ 
join  takes  an  arbitrary  number  of  arguments  and  returns  a  placeholder 
whose'  value,  ultimately,  is  the  first  of  the  arguments  to  be  given  a  value. 
In  order  to  be  useful,  of  course,  disjoin  is  ordinarily  called  with  unde¬ 
termined  placeholders  for  all  of  its  arguments.  Disjoin  is  driven  by  the 
mechanism  that  gives  values  to  placeholders  rather  than  by  any  kind  of 
polling  or  busy- waiting. 

Disjoin  is  thus  a  perfectly  adequate  implementation  of  amb  as  described 
above^.  Fair  merge  is  easily  written  using  disjoin: 

(define  (fair-merge  si  s2) 

(touch  (disjoin  (car  si)  (car  s2)))  ;  (1) 

(if  (undetermined?  (car  s2))  ;  (2) 

(cons-stream  (car  si)  (fair-merge  s2  (edr  si))) 

(cons-stream  (car  s2)  (fair-merge  (edr  s2)  si)))) 

There  are  three  subtle  (perhaps)  points  to  be  understood  about  this 
small  program.  The  line  marked  (1)  in  the  program  contains  a  touch  to 
guarantee  that  the  first  element  of  the  output  stream  is  available  before  pro¬ 
ceeding.  Fair  merge  returns  a  stream,  so  this  first  element  must  be  known 
bt'fort'  it  can  return  a  value.  But  disjoin  does  not  guarantee  that  its  value 
lias  actually  been  computed.  The  touch  will  suspend  this  computation 
until  one  of  the  two  values  becomes  available. 

The  second  marked  line  is  a  rather  inelegant  but  effective  way  to  decide 
which  element  will  be  jilaced  in  the  output  stream.  Having  guaranteed  that 
af.  least  one  of  the  initial  elements  in  the  input  streams  is  now  known  to  have 
a  value,  it  tests  a  particular  element  to  see  if  that  is  the  one  whose  value 
is  known.  A  more  powerful  variant  of  disjoin  that  returns  the  identity  of 
the  argument  which  has  acquired  a  value,  rather  than  the  value  itself,  can 
be  easily  Imilt  on  top  of  this  (jiie. 

'.Soint'  authors  usi'  sli^^htly  difri-reiit  (lofniit ions  for  amb.  implying  fairness  or  certain 
f'/rnial  m.'it  licm.'il  ii’al  jiropi'ities.  It  is  because  MultiSrheine’s  disjoin  does  not  necessarily 
^ii[iporI  lhe.se  assumptions  t.h;\t  a  new  name  was  cliosen. 
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Finally,  notice  that  the  two  recursive  calls  to  fair-merge  reverse  the 
order  of  the  two  streams  being  merged.  This  guarantees  that  fair  merge' 
will  alternate  between  the  streams  in  case  both  streams  have  values  imme¬ 
diately  available.  In  an  n-way  fair  merge  this  would  be  replaced  by  a  cyclic 
permutation  of  the  streams. 


2.6  Summary 

By  adding  two  new  features  (placeholders,  described  in  Section  2.1,  and 
a  visible  interface  to  the  garbage  collector,  described  in  Section  2.2)  to 
Scheme  we  create  a  new  language  (extended  sequential  Scheme)  which  has 
considerably  nune  expressive  power  than  the  original. 

Because  they  introduce  a  new  method  of  sharing,  placeholders  provide 
a  mean.s  of  ea.sily  embedding  logic  variables  into  LISP  (Section  2.3).  In 
addition  they  provide  a  convenient  way  to  embed  normal-order  evaluation 
into  an  otherwise  applicative-order  language  (Section  2.4).  This  allows 
recursive  data  structures  to  be  conveniently  defined,  and  provides  a  good 
deal  of  the  j^ower  of  a  normal-order  language  without  the  problems  of  either 
strictness  analysis  or  loss  of  control  over  space  utilization. 

This  power  can  be  conveniently  exploited  by  using  placeholders  in  the 
implementation  of  streams  of  data  objects  (Section  2.5).  Doing  so  solves 
two  often  troublesome  problems: 

•  Because  a  computation  is  automatically  delayed  when  a  value  it  re¬ 
quires  is  not  available,  it  is  not  necessary  to  uniformly  impose  this 
delay  on  all  uses  of  a  procedure  just  because  it  is  required  for  one 
use. 

•  As  will  be  shown  in  Section  4.3.2,  placeholders  can  be  created  to 
represent  ‘‘the  first  value  to  converge.”  This  provides  an  elegant  base 
for  implementing  McCarthy’s  amb  operator  and  the  fair-merge  of 
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This  chapter  describes  the  work  that  went  into  converting  the  extended 
sccpiential  Scheme  language  of  Chapter  2  into  a  systems  programming  lan¬ 
guage  for  a  parallel  processor.  Sections  3.1  and  3.1.1  motivate  and  de¬ 
scribe  the  global-interrupt  operation  and  synchronizer  objects  used  to 
coordinate  the  actions  of  processors  (as  opposed  to  the  placeholders  that 
coordinate  the  actions  of  tasks).  Sections  3.1.3  and  3.1.4  then  show  addi¬ 
tional  uses  for  these  extensions.  A  few  additional  procedures,  of  much  less 
interest,  are  described  for  completeness  in  Section  3.2. 

Section  3.3  discus.ses  the  design  decisions  related  to  providing  storage 
on  a  per-task  basis.  Dynamic  and  fluid  binding  are  introduced  and  com¬ 
pared  in  Sections  3.3.2  and  3.3.3,  respectively.  The  mechanism  used  by 
MultiScheme  to  implement  fluid  variables  in  a  parallel-processing  system  is 
described  in  S(’ction  3.3.4.  Finally,  Section  3.3.5  describes  several  mecha¬ 
nisms  available  without  any  further  extensions  that  can  be  used  to  provide 
storage'  intended  to  Ite  rf)m[)letely  private  to  a  task. 

Finally,  Section  3.4  introduces  MultiScheme's  task  object.  The  impor¬ 
tant  concc'pt  of  a  task  goal,  which  emibles  the  garbage  collector  to  detect 
tasks  that  tire  no  longer  neede'd,  is  described  in  Section  3.5.  .Again,  no 
language-  e-xte’nsie)!!  is  ne'e-ele-el  te)  siipitort  tasks,  but  the  separation  eif  tasks 
and  plae-ehedele’is  is  e-xitlene-el  in  MtiltiSe'heme  for  the  first  time. 
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3.1  Processor  Coordination 

MultiSi'lu'ine  provides  two  methods  (globcil  interrupts  and  synehroni/xTs) 
for  coordinating  the  activities  of  processors.  Unlike  placeholders,  which 
coordinate  the  acti\ity  of  ta.<ks  (logical  ]>rocesses  generated  Vjy  running 
i)rograms).  these  uvo  operations  deal  with  the  physical  processing  units 
of  the  hardware.  .\s  such,  they  are  more  often  used  by  the  MultiScheme 
system  itself  than  by  application  programs.  The  two  operations,  while  not 
necessarily  novel,  have  the  virtues  of  simplicity  and  compatibility.  They  also 
serve  as  a  base  for  higher-level  constructs  usehil  in  parts  of  the  system  not 
related  to  garbage  collection  (see  the  examples  of  Section  3.1.3  and  3.1.4). 

3.1.1  Starting  Garbage  Collection 

Me.ltiScheme  was  built  by  first  extending  the  Scheme  language  as  discussed 
in  Chapter  2.  This  system  was  tested  on  a  sequential  computer  by  using 
time- slicing  to  simulate  a  parallel  processor.  The  next  stage  of  development 
involved  moving  this  imi)lementation  to  an  actual  parallel  processor.  One  of 
the  early  problems  encountered  in  this  move  was  modifying  the  mechanism 
ux'd  tu  initicUe  a  garbage  collection  in  order  to  make  it  compatible  with  a 
parallel  processor.  In  (serial)  MIT  Scheme,  garbage  collection  is  initiated 
in  three  phtises.  using  a  system  modeled  after  a  hardware  priority  interrupt 
mechanism: 

Interrupt  Request 

During  some  operation  the  processor  notices  that  it  is  low  on  memory 
and  .sets  a  bit  requesting  a  garbage  collection  interrupt. 

Interrupt  Detect 

Th(’  interpreter  and  compiled  code  periodically  poll  the  interrupt  bits. 
.A  pending  interrupt  is  serviced  if  no  higher  level  interrupt  is  pending 
or  in  progress. 

Interrupt  Service 

Before  executing  the  next  instruction  from  the  running  program,  the 
machine  calls  a  proce<lure  supplied  by  the  Scheme  runtime  system 
(the  interrupt  handler),  in  this  case  corresponding  to  the  garbage  col- 
lecfi(}n  interrujjt. 
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In  moving  to  a  parallel-processor  hardware  base  there  was  no  need  to  mod¬ 
ify  the  basic  interrupt  mechanism  but  some  of  the  details  were  modified. 
Because  all  of  the  processors  share  a  common  address  space  for  the  heap  it 
is  essential  that  they  cease  computing*  before  the  garbage  collector  begins 
relocating  objects. 

The  system  must,  therefore,  support  some  mechanism  for  forcing  all  of 
the  processors  to  synchronize.  The  ability  to  initiate  such  global  synchro¬ 
nization  from  software  (in  addition  to  any  hardware  initiation  that  might 
be  available)  is  essential  to  several  system  services  (see  the  examples  of 
Sections  3.1.3  and  3.1.4). 

3.1.2  Language  Extensions  for  Coordination 

Only  three  modifications  to  extended  sequential  Scheme  are  used  in  Multi- 
Scheme  to  provide  coordination  among  the  processors:  the  interrupt  levels 
intersperse  global  and  local  interrupts,  the  primitive  procedure  global- 
interrupt  allows  any  processor  to  interrupt  the  others,  and  synchroniza¬ 
tion  objects  permit  all  processors  to  proceed  in  unison. 

The  new  primitive  operation  global-interrupt  is  a  software  interface 
to  the  global  interrupt  facility: 

(global- interrupt  interrupt-priority-level 
interrupt-handler 
all-clear?) 

The  interrupt-handler  is  a  procedure  that  is  executed  by  all  the 
other  processors  once  the  interrupt  is  detected  by  that  processor.  Initiating 
a  global  interrupt  is  a  system-wide  resource,  and  the  global-interrupt 
mechanism  forces  serialized  access  to  the  resource.  A  processor  receives 
permission  to  initiate  a  global  interrupt  only  when  no  interrupt  (local  or 
global)  of  a  higher  priority  is  pending.  At  that  time,  it  calls  the  all-clear? 
procedure  to  determine  whether  or  not  the  interrupt  should  actually  be  ini¬ 
tiated.  This  test  is  used,  for  example,  to  guarantee  that  a  garbage  collection 
global  interrupt  is  issued  exactly  once  even  though  the  need  for  it  may  be 
detected  independently  by  multiple  processors.  The  value  returned  by  a 

'  Mult  iScliiMiie  u.ses  a  .stop-and-copy  garbage  collection  algorithm.  The  subsequent  dis¬ 
cussion.  with  only  sliglit  modification,  applies  equally  well  to  initiating  the  space  flip  in  a 
real-time  copying  garbage  collection  algorithm. 
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call  to  global-interrupt  is  the  value  returned  by  all-clear?  so  the  pro- 
c('ssor  issuing  the  interrupt  can  determine  whether  or  not  the  interrupt  was 
actually  generated. 

The  global-interrupt  primitive  returns  control  to  the  caller  only  after 
till'  ituernipt  is  initiated  or  the  all-clear?  procedure  indicates  tluit  no 
inteniijit  should  take  place.  It  guarantees  that  all  of  the  processors  will 
stop  their  ordinary  work  as  .soon  as  they  poll  their  own  interrupt  bits,  an 
event  tluit  the  interpreter  and  compiler  force  to  occur  fairly  often.  This 
alone,  however,  is  not  sufficient  to  .solve  the  problem  of  starting  a  garbage 
collection.  Befor*'  any  jirocessor  can  begin  the  actutil  gtirbage  collection 
ojieration.  all  j^rocessors  must  be  entering  the  garbage  collection  operation; 
the  global  interrupt  mechanism  provides  a  way  of  initiating  an  action,  but 
does  not  provide  synchronization. 

Instetul.  MultiScheme  provides  a  pair  of  procedures  for  this  purpo.se: 
make-synchronizer  and  await-synchrony.  To  synchronize  all  of  the  pro¬ 
cessors.  one  jirocessor  makes  a  synchronizer  object  and  then  forces  all  of  the 
other  processors  to  call  await-synchrony  with  the  synchronizer  as  argu¬ 
ment.  W  hen  till  processors  are  waiting  for  synchrony  on  the  same  synchro¬ 
nizer  obji'ct  they  all  return  from  the  call  to  await-synchrony.  Typically, 
one  processor  tnakes  one  or  more  synchronizers  and  then  uses  global- 
interrupt  to  force  the  other  processors  to  begin  waiting  on  them. 

Wliile  separating  these  two  operations  can  cause  deadlocks  if  they  are 
used  improperly,  the  operations  do  serve  two  distinct  purposes.  The  ex- 
amitles  of  the  next  two  sections  show  how  the  separate  operations  can  be 
used  to  provide  higher-level  operations  that  are  not  as  easily  provided  if 
the  low-level  operations  are  bundled  together. 

3.1.3  Pause-Everything 

Clamen[13]  describes  an  early  investigation  into  debugging  tools  for  dealing 
with  the  parallelism  of  MultiScheme  programs.  He  identified  a  variety  of 
situations  recpiiring  a  program  to  temporarily  stop  all  other  work  on  the 
''Vstem.  ])erform  some  action,  and  then  allow  the  work  to  proceed.  In  or¬ 
der  to  inovidt'  this  ability,  he  implemented  the  original  pause-everything 
procedure.  A  new  implementation  extending  Clamen’s  original  version  is 
depicte<l  in  Figure  3.1.  simplified  for  purposes  of  explanation.  It  provides 
a  me'->,ige  passing  interface  to  an  object  rejjre.senting  the  tasks  that  were 
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available  for  execution  at  the  time  of  the  call  to  pause-everything.  This  j 

interface  is  described  in  Section  4.6  and  the  support  routines  referenced  in  j 

the  procedure  are  discussed  in  detail  in  Chapter  4. 

Pause-everything  uses  both  the  global  interrupt  mechanism  and  the 
synchronizers.  A  global  interrupt  is  necessary  to  force  the  other  processors 
to  save  their  state  and  become  idle.  The  synchronizers  are  used  to  divide 
the  work  into  two  phases. 

The  first  phase  is  initiated  by  the  call  to  global-interrupt  and  ends 
when  all  processors  have  arrived  at  the  first  synchronization  point.  The 
interrupt  guarantees  that  all  processors  except  the  one  that  called  pause- 
everything  will  begin  executing  the  code  in  interrupt-handler.  Thus 
the  other  processors  save  away  the  state  of  the  task  they  are  executing 
and  place  it  on  the  queue  of  work  to  be  performed.  They  then  wait  for 
all  processors  to  execute  (await-synchrony  drain-synch).  When  all  the 
processors  arrive  at  this  point,  the  tasks  available  for  execution  (including 
the  ones  that  were  formerly  executing)  have  been  saved  on  the  work  queue. 

The  processors  proceed  past  the  synchronization  point,  beginning  the 
second  phase.  All  but  the  initiating  processor  will  arrive  immediately  at  the 
second  rendezvous  point,  the  call  to  (await-synchrony  proceed-sync) . 

The  initiating  task,  however,  first  saves  away  the  contents  of  the  work  I 

queue  in  the  variable  queue  and  empties  the  queue.  All  the  processors  i 

again  rendezvous,  ending  the  second  phase. 

The  initiating  task  makes  the  message  accepting  object  based  on  the 
value  of  queue  (by  calling  the  procedure  make-returned-obj  ect  shown  in 
Figure  4.12  on  page  115).  This  becomes  the  value  of  the  original  call  to 
pause-everything.  The  other  processors,  however,  have  now  finished  the 
procedure  which  is  the  argument  to  saving-state.  But  saving-state 
does  not  return  to  the  procedure  that  called  it.  Instead  it  tries  to  get  work 
from  the  (now  empty)  work  cpieue.  So  at  this  point,  the  initiating  task 
is  still  running  on  the  initiating  processor.  All  the  other  processors  have 
relinquished  the  task  they  were  executing  and  are  waiting  for  more  work. 

The  system  has  "paused.” 

.A.n  interesting  detail  has  been  deliberately  omitted  in  this  description. 

What  shoidcl  happen  if,  while  other  work  is  suspended,  the  running  task 
touches  the  placeholder  associated  with  one  of  the  suspended  tasks?  While 
there  is  no  ol)\’iously  correct  answer,  the  actual  MultiScheme  system  marks 
these  suspended  tasks  paused.  Touching  a  ijlaceholder  that  is  markcfl  paused 
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(define  (pause-everything) 

(let  ((drain-synch  (make-synchronizer)) 

(proceed-synch  (make-synchronizer) ) ) 

(define  (interrupt-handler) 

(saving-state  ;  (1) 

(lambda  () 

(await-synchrony  drain-synch) 
(await-synchrony  proceed-synch) ) ) ) 
(global-interrupt  high-priority 
interrupt-handler  (lambda  ()  #T)) 
(await-synchrony  drain-synch) 

(let  ((queue  (drain-work-queue)))  ;  (2) 

(await-synchrony  proceed-synch) 
(make-returned-object  queue))))  ;  (3) 


Notes: 

I .  Saving-State  stores  away  the  state  of  the  task  currently  executing  on  this  processor 
and  places  it  on  the  work  queue.  It  then  calls  the  procedure  which  is  its  only 
argument.  Saving-State  never  returns  to  its  caller:  it  looks  for  work  from  the 
queue  when  the  argument  procedure  is  finished.  See  the  discussion  in  Section  4.2.3 
and  the  code  in  Figure  4.8  on  page  102. 

2  Drain-work-queue  empties  the  queue  of  tasks  awaiting  processors  and  returns  a 
list  of  the  tasks  removed.  See  the  description  of  drain-work-queue  in  Section  4.1.3. 

3  Make-returned-object  creates  the  message-accepting  object  that  is  the  result  of 
a  rail  to  pause-everything.  See  the  discussion  of  make-returned-object  in  Sec¬ 
tion  4.6  and  the  code  in  Figure  4.12  on  page  115. 

Figure  3.1;  Simplified  Code  for  Pause-Everything 
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is  handled  by  the  scheduler  of  Chapter  4  in  exactly  the  same  way  the 
scheduler  handles  touching  a  placeholder  marked  delayed:  the  associated 
task  will  be  reactivated.  One  of  the  advantages  of  writing  the  scheduler  in 
MultiScheme  is  that  such  decisions  can  be  easily  changed.  For  example,  it 
is  quite  easy  to  add  a  "scheduler  hook”  to  allow  users  to  specify  their  own 
way  of  handling  this  situation. 

3.1.4  Within-Task 

The  final  example  of  processor  coordination  is  a  critical  part  of  the  user 
interface,  allowing  the  user  to  interact  with  previously  started  tasks.  The 
primary  use  of  this  facility  occurs  in  the  top  level  interaction  between  a 
user  and  the  MultiScheme  system.  In  MIT  Scheme,  a  user  can  at  any  time 
interrupt  execution  of  a  program  and  create  an  interaction  environment 
within  the  state  of  the  program  at  the  time  the  interrupt  was  serviced  (by 
issuing  a  "breakpoint  interrupt”).  A  user  can  also  interrupt  the  program 
and  force  it  to  throw  back  to  an  earlier  interaction  environment,  effectively 
aborting  the  current  computation 

The  direct  extension  of  this  into  MultiScheme  would  allow  a  user  to 
interrupt  the  system  and  interact  with  any  one  of  the  tasks  in  the  system^ 
at  the  time  of  the  interrupt.  Because  a  task  has  state  information  visible 
to  the  j.rogrammer  (see  Section  3.3)  it  is  important  that  the  correct  task 
actually  interact  with  the  user.  .A.s  a  result,  there  must  be  some  way  to 
force  a  selected  task  to  call  the  procedure  which  implements  the  interaction 
environment  or  invoke  a  continuation  that  aborts  the  current  computation. 

Within-task  is  designed  to  facilitate  this  and  other  similar  operations. 
Some  of  Clamen's  debugging  tools,  for  example,  rely  on  within-task  in 
order  to  rei)ort  the  progress  of  a  program  back  to  the  user.  A  simplified 
version  of  this  procedure  is  shown  in  Figure  3.2.  It  expects  two  arguments, 
a  ta.sk  and  a  thunk,  and  forces  the  task  to  execute  the  thunk  before  con¬ 
tinuing  with  whatever  processing  it  is  currently  doing.  The  operation  of 
this  procedure  is  ijitertwined  with  the  scheduler  (so  the  details  are  deferred 
to  C'liapter  4)  but  th(’  overview  demonstrates  a  different  use  of  the  global 
interrupt  luechanisin. 

“  I  h''  usi-r  iiitcrfacc  iioriiially  used  for  .MuIti.Scheme  does  not  allow  the  user  to  select  a 
partn  ular  t;usk  for  interaction.  Rather,  it  chooses  randomly  from  the  curreiitiy  e.xecuting 
t  a.^k 
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(define  (within-task  task  thunk) 

(with-task-locked  task 

(lambda  (task-still-runnable?) 

(if  (eq?  (task. status  task)  ’RUNNING) 

(begin 

(set-task . status !  task  ’WITHIN-TASK) 

(set-task . code !  task  thunk) 

(global-interrupt  high-priority 
(lambda  ()  (if  (eq?  (current-task)  task)  (reschedule))) 
(lambda  ()  #T))) 

...)))) 


Notes:  Di'scripiiou  of  the  ta.sk  data  structure  as  well  a.s  the  utility  routines  with-task- 
locked,  current-task,  and  reschedule  are  provided  in  Chapter  d. 

Figure  3.2:  Siniitlified  Code  for  Within-Task 


:hin-task  works  by  testing  whether  the  task  is  cnrreutly  executing 
'  of  the  procosoi'.  If  so,  it  marks  the  task  indicating  that  it  must  be 
to  exectfe  tlie  v]>ecihi‘d  Code  ;ind  then  initiates  an  interrupt  on  all 
^o'  -,  foi'  U.y,  'l.o  oi.o  re.nning  the  chosen  task  to  reschedule  the  task 
■ n  c.  Ti.'  ;  >; .  H-<  d,ur<'  reschedule,  described  in  Section  4.4, 
'  'U'  1.  a  way  th-at  the  "pecified  code  is  executetl 

b  -■  ■  b  ; ,  -  :,t  ’Ut  at  k  Ui, 

■  ,  '..i  tlu^  one  dot's  not  nt'etl  any 

:  ■  I ,  4  the  mti'rruitt  is  mert'ly 
:  ‘  :  w;'!.oiit  Ituowing  in  tidvtinct' 

■  •  .pted  from  t heir  dut it's  to 

•  ;..;:.'  o;ati  ly  and  mtlependently. 


.‘i.'J  Miii<-r  f  \»«  n^ion- 


oi  r  exirnded  setpien- 
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•  Programs  can  ask  the  number  of  Scheme  interpreters  in  the  system 
(using  N-Interpreters)  anci  subdivide  work  accordingly.  They  can 
also  ask  which  interpreter  they  are  currently  running  on  (using  My- 
Interpreter-Number).  MultiScheme  also  provides  a  My-Processor- 
Number  for  use  in  reporting  hardware  problems^. 

•  MultiScheme  provides  an  abstraction  of  an  underlying  task  distri¬ 
bution  mechanism,  described  in  Chapter  4  along  with  the  details  of 
implementing  the  MultiScheme  scheduler.  User  programs  are  not 
expected  to  deal  with  this  set  of  primitives  unless  they  modify  the 
scheduler  to  provide  specialized  scheduling  policies. 

•  MIT  Scheme's  mutation  operations  (set!,  set-carl,  etc.)  were  al¬ 
ready  defined  to  return  the  previous  value  stored  in  the  cell  they 
modify.  In  MultiScheme  these  operations  are  performed  atomically 
by  the  interpreter.  The  compiler  may  detect  the  fact  that  the  value 
is  not  used  and  convert  them  into  pure  write  operations  instead  of 
atomic  swaps. 

•  MultiScheme  supports  operations  derived  from  Multilisp[26]  for  per¬ 
forming  atomic  conditional  modify  operations.  In  MultiScheme,  these 
operations  are  set-car-if-eq? ! ,  etc. 


3.3  Per- Task  Storage 

in  a  system  that  suppcn  ts  concurrently  executing  tasks  there  is  some  amount 
of  information  which  constitutes  the  state  of  the  task  from  the  point  of  view 
of  the  underlying  system.  But  in  addition  to  this  system-imposed  state,  the 
program  structun'  itself  may  require  storage  on  a  per-task  basis  (see,  for 
example,  the  Uniform  System  example  of  Section  5.4.3).  Providing  this 
storage  is  closf'ly  related  to  issties  of  variable  naming,  and  MultiScheme 
has  extendf’d  the  fluid  variables  of  MIT  Scheme  to  provide  per-task  storage. 

Section  3.3.1  describes  Scheme's  mechanism  for  providing  localized  stor¬ 
age  in  a  sequential  program  through  the  use  of  lexically  scoped  variables 

■’  "liiti  r|irct(‘r  iminlx'r.s"  are  a.ssigiKxJ  .se<iueiit iaily  from  0  tlirougli  N-Interpreters— 1 . 

■  i’roc-essor  iiiinilu  rs”  are  a.ssigiie<l  by  the  underlying  hardware  and  are  not  neces.sarily 
seipietit  ial . 
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.1  ’I  '  faiuiliar  as  '‘hlork  structure"  iii  other  languages.  One 

ti'._  a  i.^iy  eiiri iiiutered  problem  with  this  structure,  however,  is  the  <lif- 
:'.hy  Ilf  allowing  a  singh'  namt’  to  <leiiote  different  values  bast'd  on  the 
Control  riow  of  the  prt>gram.  the  core  of  the  difficulty  encountered  witli 
pi'!  ra,--k  vaii.ables  in  MultiScheine.  Section  3.3.2  describes  the  tradition.al 
Li'p  <lynaniic  binding  approacii  to  solving  this  problem,  whih.'  Section  3.3  3 
di'-i  iibes  the  fluid  variables  used  in  Scheme.  Section  3.3.4  describes  the 
i!n[)lementation  changes  required  to  provide  fluid  varial)les  in  a  parallel 
system,  and  Section  3.3.5  concludes  by  pointing  out  a  set  of  julditional  oj)- 
tious  that  might  be  pursued  if  the  block  structure  of  these  per-task  variables 
is  not  an  important  concern. 

3.3.1  Packaging  and  Lexical  Scoping 

The  problem  of  controlling  a  program’s  name  space  was  one  of  the  im¬ 
portant  motivations  in  the  original  design  of  Scheme.  This  led  Scheme  to 
depart  from  the  dynamic  binding  of  variables,  which  had  been  one  of  the 
inimary  underpinnings  of  Lisp  for  over  a  decade,  in  favor  of  lexical  scoping 
of  variu!)le  names.  One  of  the  hallmarks  of  MIT  Scheme  has  been  its  insis- 
t<'nce  on  using  only  le.xical  scoping  for  variable  reference.  In  this  regard  it  is 
like  the  .41gol  family  of  languages.  Because  Scheme  also  includes  operations 
with  side-effects,  programs  written  in  Scheme  are  analyzed  using  an  envi¬ 
ronment  model  ill  which  procedures  contain  both  the  code  for  executing  the 
procedure  and  the  environment  in  which  free  variables  are  to  be  resolved'*. 

One  well-known  and  desirable  property  of  lexical  scoping  stems  from 
the  fact  that  the  scope  of  a  variable  is  textually  localized.  Thus  a  compiler 
for  a  lexically  scoped  language  can  make  deductions  about  the  ways  in 
which  I'ariables  are  referr'uced  even  when  presented  with  small  parts  of  the 
system.  Unlik<’  a  compiler  for  a  dynamically  scoped  language,  it  does  not 
need  to  see  an  entire  program  to  know  that  a  ct'rtain  variabh-'s  Vciluc  never 
changt's.  or  that  it  is  unreferenced.  The  compiler  need  only  sec  the  code  that 
comprises  the  scope  of  the  variable.  Thus  separate  compilation  of  modules 
can  be  performed  with  optimization  of  variable  references  in  a  h'xically 
sccjped  language  witluait  imposing  unnatural  rides  about  referencing  free 

'  I  lie  coinpiliT  IS  'iflt  ii  al>le  to  that  ,'Oiu<'  of  this  iiiforiuatioti  is  not  needl'd  at 

run  tune  and  n,  free  to  clioo.se  more  e(heii-nt  rejiresent at lon.s  litvsed  on  these  deductions. 
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variables,  or  introducing  unwarranted  assumptions  about  the  behavior  of 
components  of  the  program  external  to  the  compilation  unit. 

This  same  ability  to  subdivide  the  code  aids  programmers  and  designers 
in  building  and  evolving  large  systems.  The  scoping,  or  block  structure, 
j)rovides  a  simple  and  effective  mechanism  for  isolating  independent  parts 
of  a  system  from  one  another.  At  the  same  time,  it  provides  an  easily 
used  and  easily  detected  form  of  communication  through  the  use  of  free 
variables.  Combining  these  in  different  ways  provides  a  great  deal  of  power 
aiul  flexibility.  One  common  pattern  found  in  MIT  Scheme  comes  from 
the  ability  to  treat  environments  as  first-class  objects.  With  this  ability  it 
becomes  possible  to  create  a  package  of  related  procedures,  similar  in  spirit 
to  a  Clu  cluster [38]. 

A  Scheme  package  is  a  single  environment  frame  which  contains  a  num¬ 
ber  of  variables  that  are  global  to  the  package.  Most  packages  include  in 
these  "package  variables”  a  number  of  procedures  that  perform  activities 
using  the  variables  to  guide  their  behavior.  Some  of  these  procedures,  con¬ 
taining  the  package  as  the  environment  to  use  for  resolving  free  variable 
references,  are  also  exported  from  the  package  to  serve  as  interface  proce¬ 
dures  with  the  rest  of  the  Scheme  system.  Figure  3.3  shows  a  typical  piece 
of  MIT  Scheme  code  to  construct  a  hypothetical  package  used  for  printing 
numbers.  This  has  one  explicitly  declared  package  variable  (radix)  in  ad¬ 
dition  to  all  three  of  the  procedures  that  are  created  using  define.  Two 
of  these  three  procedures  are  exported  to  the  external  environment  (using 
the  access  special  form  of  MIT  Scheme),  and  one  of  them  is  “renamed”  in 
the  process  (i.e.  it  is  referenced  by  one  name  within  the  package  and  by  a 
different  name  from  outside  the  package). 

3.3.2  Dynamic  Binding 

The  package  introduced  in  Section  3.3.1  also  demonstrates  a  common  dif¬ 
ficulty  with  this  organization.  Suppose  we  discover  a  frequent  need  to  ex- 
rcutc  an  existing  program  that  prints  some  numbers  using  print-number, 
Imt  with  outpiit  in  octal  or  hexadecimal  rather  than  decimal.  Our  first 
inclination  might  be  to  write  a  new  interface  procedure  as  part  of  the  pack¬ 
age: 
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(define  number-package 
(let  ((radix  10)) 

(define  (number->list-of -digits  number) 

...) 

(define  (print-number  number  stream) 

(for-each 

(lambda  (digit)  (display  digit  stream)) 
(number->list-of-digits  number))) 

(define  (cheinge-radix !  new-radix) 

(let  ((old-radix  radix)) 

(set!  radix  new-radix) 

old-radix))  ;  Return  old  radix 

(the-environment)))  ;  (1) 

; ;  Now  export  the  interface  procedures 
(define  print-number 

(access  print-number  number-package))  ;  (2) 

(define  set-print-radix! 

(access  change-radix!  number-package)) 


Notes: 

1.  The-Enviroiunent  supplies  the  current  environment  frame  as  an  ordinary  Scheme 
object. 

2.  Access  performs  a  (lexical)  variable  lookup  in  a  specified  environment  frame. 

Figure  3.3:  A  Simple  Package 
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(define  (with-new-radix  procedure  new-radix) 

(let  ((old-radix  (set-print-radix!  new-radix))) 

(let  ((result  (procedure))) 

(set-print-radix!  old-radix) 
result) ) ) 

We  change  the  radix  and  remember  the  old  radix.  Next  we  call  the 
procedure,  and  finally  set  the  radix  back  to  its  old  value.  This  works 
unless  somehow  the  procedure  we  are  calling  manages  to  return  back  past 
the  call  to  with-new-radix  without  executing  the  code  to  set  the  radix 
back.  This  can  only  happen  if  during  the  execution  of  that  procedure  a 
continuation  made  earlier  in  the  computation  is  used.  This  might  at  first 
appear  to  be  a  fairly  unlikely  occurrence;  but  it  is  precisely  what  happens 
if  the  user  interrupts  the  computation  using  within-task  as  described  in 
Section  3.1.4. 

By  allowing  first-class  continuations  into  the  language,  we  have  obliged 
MIT  Scheme  to  provide  some  mechanism  for  handling  this  kind  of  multi¬ 
level  return.  Many  languages  provide  an  “exception  handling  mechanism” 
to  handle  this  situation,  and  MIT  Scheme  provides  a  very  general  mecha¬ 
nism  that  serves  this  purpose,  described  in  Appendix  D.  The  problem  here, 
however,  is  a  simple  one  that  illustrates  a  problem  with  lexical  scoping  com¬ 
pared  to  dynamic  binding.  In  a  dynamic  binding  discipline  the  values  of 
variables  are  located  with  respect  to  the  execution  stack  at  the  time  of  the 
reference.  So,  if  we  used  dynamic  binding  for  the  value  of  radix  we  could 
safely  use  the  following  piece  of  code: 

(define  (with-new-radix  procedure  radix) 

(procedure) ) 

Introducing  “just  a  little”  dynamic  binding  into  Scheme  requires  the 
careful  examination  and  resolution  of  a  number  of  potential  problems  be¬ 
tween  the  interactive  ('iivironment  and  tools  with  a  more  static  view  of 
I)rograms.  such  as  a  compiler.  The  goal  of  the  MIT  Scheme  interactive 
environment  is  to  sujjport  the  very  simple  model  of  a  user  positioning  an 
interaction  loop  inside  an  environment  corresponding  to  a  specific  proce¬ 
dure  invocation.  This  allows  users  to  enter  an  arbitrary  Scheme  expression 
and  find  its  value  at  that  po.sition  within  the  program.  But  in  order  to 
do  this,  there  must  be  some  way  of  knowing  which  free  variables  in  tlu' 
expression  are  to  l)e  referenced  using  dynamic  binding  (and  hence  use  the 
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call  stack)  as  oppos*'!!  to  those  that  are  to  be  refeTeiice<l  l<-xically  (using 
the  chain  of  frames  of  definition).  In  g«“neral  this  information  must  also  be 
known  at  compilation  tinu''’,  and  might  be  introduced  in  a  variety  of  ways. 

1.  The  source  text  coidd  r<'<inir(“  explicit  dc'chirations  stating  which  way 
to  treat  variables  for  which  no  binding  is  found  at  comj)ile  time.  Un 
fortunately,  at  run  tim<“  tliert"  is  no  information  available  that  six-cifies 
what  parts  of  a  program  wer<*  compile<l  as  a  single  unit,  or  which  dec 
larations  wen'  in  effect  for  wh:it  vari.'ibles.  As  a  result  it  is  not  ])ossil)le 
to  determine  which  references  <'ntered  by  the  user  were  to  vjiriables 
rletermined  by  the  compiler  to  be  dynamic  as  opposed  to  lexical.  Of 
course,  tlu'  comi)iler  coidd  leave  around  such  information,  but  this 
jilaces  an  otherwise  unn«'c<'ssary  burth'u  on  the  compiler  and  runtime 
system. 

2.  A  default  assumption  can  be  us<'d,  as  in  ZetaLisp.  In  this  cas<'  ref¬ 
erencing  ;uiy  variable  for  which  no  binding  is  provided  by  the  com- 
piltition  unit  returns  the  dynamic  binding  of  the  varialile*’.  But  this 
has  precisely  tin'  same  problems  as  the  earlier  sohition,  since  this 
information  is  also  not  available  at  run  time. 

3.  A  new  siiecial  form  could  be  created  for  introducing  dynamic,  bind¬ 
ings.  This  is  an  interesting  compromisi!  and  is  the  basis  for  tlu'  actual 
choice  made  in  MIT  Scheme  and  extended  in  MultiSclieme.  This 
o[)tion  is  discussed  in  Section  3.3.3,  below. 

4.  Special  forms  could  b<'  introduced  for  both  referencing  and  intro¬ 
ducing  a  dynamic  binding  of  a  variable.  This  option  is  so  burden¬ 
some  that  it  would  mak**  the  use  of  dynamic  variabh^s  virtually  non¬ 
existent.  It  also  opens  up  the  possibility  of  i)rogram  l)ugs  arising  from 
inadvertently  referencing  the  dynamic  l>inding  of  a  variable  wIk'u  th<' 
lexic.'il  value  is  desired,  and  vice  versa. 

'by  “roinpilation  titiic”  I  mean  the  lime  wlieii  Scheme  source  code  is  converted  into  an 
e.xi  rutalde  program,  riiiis,  compilation  occurs  even  when  Scheme  expressions  are  typed 
III  or  loaded  lliterac  lively 

'■'I  he  actual  rules  of  ( ’ommonlasp,  on  the  other  hand,  are  (piite  comiilicaleil  here,  and 
even  permit  the  [(ossihilily  of  changing  <it  niv  ttmr  tlie  type  of  hinding  to  he  u.sed  'I'o  the 
.iiithor's  knowledge  no  implementation  of  Commonhisji  actually  supjiorts  this  option. 
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MIT  SrhciiK'  iidoptcd  iin  int('inH'diat«‘  solution  to  this  i)rol)lcin.  Because  of 
the  difficulties  of  htiildiiig  an  interactive  syst<‘in  consistent  with  two  distinct 
\  ;\i  i:ihle  referencinj;;  and  binding  inechanisins,  it  provides  a  iiK'chanisin  that 
iloes  not  affect  the  h'xical  nature  of  bindings  in  the  language.  Instead,  it 
provides  a  mechanism  allowing  the  existing  h'xical  lookiij)  to  effectively  r<'f- 
erenc<-  different  locations  at  diff<“r<*nt  times,  based  on  the  dynamic  exectition 
of  the  program. 

3.3.3  Fluid  Variables 

Rather  than  introdiua-  dynamic  binding,  MIT  Scheme  provides  the  special 
form  fluid-let,  originally  introduced  by  Stissman,  to  alter  the  value  of  an 
existing  variable  for  use  during  the  execution  (time)  of  a  particular  segment 
of  code.  All  variable  references  remain  completely  lexical,  but  the  location.'^ 
wher<‘  these  “Huid  variable.s”  are  stored  is  altered.  Syntactically,  fluid- 
let  is  exactly  tlu'  same  as  let.  The  solution  to  the  earlier  i)robl('m  can  be 
<‘Xi)ressed  by  adding  the  following  i)rocedure  to  the  package: 

(define  (with-new-radix  procedure  neu-radix) 

(fluid-let  ((radix  neu-radix)) 

(procedure) ) ) 

Fluid-let  has  a  slightly  unusual  semantics.  It  docs  not  introduce'  a 
new  binding  (as  doe.'s  let),  although  the  syntactic  terminology  refers  to 
the  varial)les  in  both  ca.s<'s  as  a  “binding  list.”.  Rather,  it  converts  the 
existing  bimlings  into  what  ar<'  termed  fluid  variables  (hence  th('  name). 
Til!  execution  of  a  fluid-let  expre.s.sion  sei)arates  the  dymunic  execution 
of  a  [)rogra,m  into  two  parts.  Before  entering  the  body  of  the  fluid-let, 
and  after  exiting,  the  varitibles  spr'cified  in  the  binding  list  (in  this  case 
radix)  refer  to  a  certain  .set  of  locations  in  the  store.  While’  execution 
is  within  the  body,  however,  the-y  refer  to  a  new  set  of  locations  and  the- 
old  locations  are'  inae'e’e’ssible’.  The*  ne’w  locatie)ns  <ire  cre’ate’el  whe’ii  e‘e)ntre)l 
initially  e'lite’is  the’  be)ely  e)f  the’  fluid-let,  anel  the’y  are  initi.dize’el  te)  the’ 
values  spe’e-ifie’d  in  the’  bineling  list  (in  this  ca.se'  the-  value’  e)f  the’  e’xpre’ssion 
neu-radix ). 

Bee'ause’  eif  the’  e’xiste’iie’e’  eff  first-edass  cemtinuiitiems  in  the’  language’,  the 
iiiipleme'iit ation  eif  fluid-let  is  ne)t  trivial.  In  MIT  Sche’ine’  it  is  baseel 
on  the  mechanism  ele’scribe’el  in  Ap])e’nelix  D  whiedi  guarantee’s  that  any 
attempt  to  ereate’  a  e'e)ntinuatie)n  ehiring  the’  e’xe’e’utie)n  e)f  the-  bexly  eif  the- 
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fluid-let  (■oircctly  cajitiires  th<>  locations  of  tlu'  Hiii<l  varial>!<‘s.  W'licn 
'-m  il  a  coiitiuuation  is  us(><l,  wlictli<T  from  inside  or  outside  the  execution 
of  the  hody,  those  locations  ar<‘  restored. 

Fluid  v:iri;d)les  .are  used  in  th<‘  MIT  Sclu'nu'  runtinu'  system  for  imiiii- 
t.uuiufj;  the  stat('  of  ii  I:uy!;e  numi)er  of  packages  witiiin  the  system.  These 
vat ia! lies  include  the  primary  input  and  ovitjuit  str<':ims,  current  syiitiixer 
tahlo  for  parsing  progi'iims,  iuforimition  about  tlu'  curnuit  (uror  or  luautk- 
poiiit  cout<‘xt,  and  recursivt*  state  of  the  compiler.  They  provide  :i  coiive 
nieiit  euciiiisulation  of  tin  important  construct  whose  power  is  needed  in  :iny 
huge  system.  The  use  of  fluid  vari.ahles  has  completely  repLaced  dymimic 
liinding  witliiii  MIT  Scheme. 

3.3.4  Fluid  Variables  in  a  Parallel  System 

I,  iifort uuiitely,  lioth  the  seimmtics  and  the  m(H‘h:inism  list'd  to  implement 
fluid-let  in  .MIT  Scheme  run  into  prohlenrs  wlien  the  systt'm  is  extendeii 
to  pjuallel  ex('(aitiou.  Unlike  a  st'tpK'ntial  system  where  the  ('utire  system 
is  eiih(  r  luside  or  outside  th<;  dynamic  extt'iit  of  th('  hody  of  a  fluid-let 
ft  mu.  Mult  iScheme  ;is  <i  systt'm  can  Ix'  sinmltant'ously  inside  and  outsidt'  th(' 
i)o(!y.  Fui  tiiermore,  it  can  instantaneously  he  inside  of  many  bodies  that 
change  the  locations  of  the  same  variables.  Thus  each  task  must  somehow 
(auitjiiii  the  information  needed  to  locate  the  particuhar  cell  where  the  value 
of  each  of  its  tluid  variables  is  currently  stored. 

.MulriScheme  re.solves  thi.s  situation  by  providing  a  run  time  mechanism 
fluit  lu.'ii  ks  individutd  entries  in  an  environment  franu'  for  special  lumdling 
by  the  wuialile  looktip  aiul  side-effect  (set!)  code.  Sinct'  the  MIT  Schi'ine 
intei pretcr  ;dre;idy  exiimines  the  values  of  a  varitible  th:it  is  ri'ferenct'd  or 
modified  to  detect  unoound  and  unassigne<l  varitdile  errors,  this  same  test 
was  I'xtended  into  a  more  flexible  in<;chanism.  A  type  code  (called  trap  on 
reference  or  trap  for  short)  was  assigned  for  this  juirjiose.  Every  interpreted 
variable  reference  or  set  I  ojx'ratioii  examines  the  conti'iits  of  the  (lexically 
locared)  environment  frame  to  check  for  sjiecial  handling.  If  this  trap  code 
1-  found  the  slot  also  contains  the  information  ix'cessary  to  complete  th«‘ 
n  f  ri  la  i'  In  many  cases  the  comirih'r  <'an  omit  this  check,  since  it  can  Ix' 
d<  fo[ mined  sfaticallv  (at  c<;mpile  t'linr)  tr>  Ix'  unrx'cessary. 

In  older  to  iinj)lenient  fluid-let  the  traj)  code  is  associated  with  a 
inall  bloi  k  of  data.  This  bhx'k  includr's  a  fhag  indic.ating  tlnit  the  variable 
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in  (lucstioii  is  fluid  as  opposed  to  unbound  or  unassigned  or  soin<“  user- 
sp('rified  typ('.  It  also  contains  the  cell  where  the  value  of  the  fluid  varial)le 
is  storc'd  wlu'ii  outside  of  the  influence  of  any  fluid-let  form.  Further- 
mort',  associated  with  each  task  is  a  fluid  binding  list,  a  standard  Lisp  A-List 
mapping  trap  objects  to  cells  for  the  value  of  the  coiresponding  fluid  vari- 
abl('.  In  order  to  allow  the  introduction  of  a  future  form  within  tlie  code 
('x<'cuted  as  part  of  the  body  of  a  fluid-let  to  function  as  it  would  in  MIT 
Scheme,  a  newly  created  task  inherits  the  fluid  binding  list  of  the  task  that 
spawns  it,  and  it  can  therefore  communicate  with  that  task  and  its  own 
"siblings”  using  the  cells  (and  the  fluid  variables  they  represent)  that  are 
part  of  that  list.  Finally,  a  continuation  also  contains  the  fluid  binding  list 
that  was  in  existence  when  it  was  created,  and  invoking  that  continuation 
in  the  normal  manner  restores  those  fluid  bindings. 

To  summarize  the  mechanism  that  supports  fluid-let  in  MultiScheme: 

•  When  a  fluid-let  form  is  entered,  each  of  the  variables  in  the  bind¬ 
ing  list  is  located  using  ordinary  lexical  lookup. 

-  If  the  variable  is  not  yet  marked  as  fluid  (by  the  presence  of  a 
trap  type  code  in  the  value  cell)  create  a  new  trap  object  that 
contains  the  current  contents  of  the  value  cell  as  the  contents  of 
the  “outside”  cell.  Store  this  object  into  the  lexical  value  cell. 

-  In  either  case,  make  a  new  entry  on  the  task’s  fluid  binding  list. 
This  new  entry  contains  the  trap  object  as  key  and  the  value  of 
the  expn'ssion  in  the  fluid-let  binding  list  as  value. 

•  On  every  varial)le  refen.-nce  or  set!  opercution,  look  up  the  variable 
u.sing  ordinary  lexical  lookup.  If  the  value  of  the  variable  is  a  trap 
object  for  a  fluid  variable,  look  on  the  task’s  fluid  binding  list. 

-  If  the  trap  ol)ject  is  found  in  the  fluid  binding  list,  then  the 
corresponding  value  cell  is  used  in  place  of  the  cell  in  the  lexical 
environment  frame. 

-  If  the  trap  ol)ject  is  not  on  the  fluid  binding  list,  then  the  “out¬ 
side"  cell  of  the  trap  ol)ject  is  u.sed  in  place  of  the  cell  in  the 
lexical  environment  frame. 

•  W1  icn  a  new  task  is  created,  it  inherits  the  fluid  binding  list  of  the 
creating  task.  The  initial  task  luis  an  einjity  fluid  binding  list. 
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•  When  a  continuation  is  created,  it  contains  the  current  fluid  binding 
list  as  part  of  the  saved  computation  state. 

•  When  a  continuation  is  applied  to  an  argument  (i.e.  when  a  task 
throws  to  the  continuation),  the  fluid  binding  list  from  the  continu¬ 
ation  becomes  the  task’s  fluid  binding  list. 

Because  fluid  variables  ari'  referenced  only  rarely  and  the  fluid  binding  lists 
tend  to  remain  fairly  short  this  mechanism  has  proven  effective  in  Multi- 
Schi'ine.  The  mechanism  is  closely  related  to  the  deep  Vanding  mechanism 
originally  used  to  implement  dynamic  binding  in  Lisp,  and  suffers  from  the 
same  jiroblerns.  If  the  fluid  binding  lists  become  long  (typically  because  a 
few  variables  are  being  bound  by  a  recursive  procedure)  then  references  to 
early  entries  becomes  slow^.  For  this  reason,  most  Lisps  now  implement 
dynamic  binding  using  a  technique  known  as  shallow  binding  that  multi- 
ple.xes  the  value  cell  within  the  environment  directly.  An  implementation 
of  fluid-let  using  shallow  binding  was  considered  for  use  in  MultiScheme, 
bvit  was  rejected  because  it  would  dramatically  increase  the  time  required 
to  switch  tasks. 

3.3.5  Task-Private  Storage 

Fluid  variables  provide  an  excellent  way  for  tasks  to  create  their  own  stor¬ 
age.  Fluid-let  can  be  used  to  provide  a  value  for  a  variable  without 
(dtering  the  scx)pe  of  the  variable.  Yet  its  accessibility  is  limited  to  the  tjisk 
that  evaluated  the  fluid-let  and  any  sub-tasks  it  spawns  within  the  body. 
Furthermore,  since  the  mechanism  is  supported  by  a  uniform  variable  ref- 
<'rence  meclianism  there  is  no  need  to  modify  existing  code  to  cope  with 
the  fact  that  some  of  its  free  variables  may  become  fluid  variables  at  any 
tiiiK'  (hiring  the  execution  of  the  program.  The  mechanism  is  inherently 
dynamic,  not  static. 

Yet  this  very  flexiljility  can  be  a  jmoblern.  The  speed  of  a  variable 
reference  is  governed  by  the  usual  issues  relating  to  lexical  lookup.  For 
fluid  variables  this  time  must  be  augmented  by  the  time  retpiired  for  a  deep 
^('arch  of  the  task’s  fluid  binding  list.  If  a  task  references  a  variable  that 

'  l?y  proviiling  a  cache  for  recently  referenced  (liiid  variables  (flushed  when  a  throw 
occurs  and  wlien  a  fluid-let  body  is  e.xited),  the  cost  of  referencing  fluid  variables  can 
often  be  significantly  reduced. 
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has  a  fiiiid  hiiuliiig  l)y  soiiif  other  task,  yet  is  not  itself  within  the  body 
of  any  fluid-let  for  that  variable,  it  must  find  the  “outside”  binding 
of  that  variable.  Unfort \mately  this  requires  searching  the  task’s  entire 
fluid  l)iiidiiig  list  to  discover  that  the  task  has  no  such  binding.  Then, 
of  course,  the  correct  binding  slot  is  easily  located  from  the  trap  object 
which  was  originally  located.  Thus  the  time  for  referencing  such  variables 
is  proportional  to  the  length  of  the  task’s  fluid  binding  list  (but  see  the 
footnote  on  page  74).  There  is,  therefore,  a  desire  to  keep  this  list  as  short 
as  possible. 

.4t  the  same  time,  there  is  always  a  need  for  a  task  to  contain  data 
relevant  only  to  its  own  workings  and  for  which  the  ability  to  share  that 
data  with  other  tasks  is  minimal.  For  example,  in  MultiScheme  running 
on  the  BBN  Butterfly,  each  task  has  its  own  I/O  stream  that  it  uses  by 
ch'faidt.  While  the  task  itself  must  be  able  to  reference  these  streams  no 
application  has  yet  been  found  in  which  any  other  task  need  be  concerned 
with  them.  This  task  state  can  be  supported  directly  using  the  fluid  vari- 
ablc's  mechanism  by  placing  a  fluid-let  into  the  code  in  the  scheduler 
for  creating  tasks.  This  guarantees  that  every  task  has  its  own  location  in 
which  to  store  the  streams  when  they  are  needed. 

While  this  solution  works,  it  guarantees  that  each  task’s  fluid  binding 
list  contains  an  entry  for  each  of  these  task  state  variables.  Furthermore, 
since  eacli  task  inherits  the  fluid  binding  list  of  the  task  that  spawned  it, 
it  contains  an  entry  for  each  of  these  variables  for  each  task  in  the  chain  of 
par('nt  tasks  back  to  the  initial  task  created  by  the  system.  Thus  the  very 
act  of  creating  a  task  will  force  that  task  to  take  a  longer  time  to  reference 
fluid  variables  than  the  task  that  created  it.  This  observation  is  really  just 
the  same  one  that  initiated  the  conversion  from  deep  binding  to  shallow 
l)inding  in  traditional  Lisp  systems. 

This  difficulty  is  a  very  real  one  in  MultiScheme,  but  there  has  been  no 
single  solution.  Instead,  the  existing  mechanisms  support  three  different 
solutions  employed  for  various  parts  of  the  task  state. 

1.  For  items  that  must  be  referenced  by  the  lowest  levels  of  the  sys¬ 
tem  and  possibly  the  microcode  of  a  machine,  the  task  object  itself 
contains  a  (fixed)  number  of  .slots.  Since  the  task  object  can  be  ma¬ 
nipulated  from  within  MultiScheme,  it  is  possible  to  view  the  task 
object  itself  as  a  data  structure  containing  certain  pieces  of  state. 
This,  of  (-(jurse,  requires  that  the  state  variables  care  fixed  and  known 
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in  advance.  Furthermore,  it  requires  that  any  code  that  manipulates 
this  state  be  aware  that  it  is  stored  in  a  particular  fashion.  The  task 
data  structure  is  described,  along  with  a  number  of  routines  that  deal 
with  it,  in  Section  4.1.2. 

2.  One  of  the  slots  in  the  task  structure  can  contain  a  standard  lexical 
environment  frame.  This  frame  contains  the  extensible  portion  of  the 
task  state.  This  solves  the  problem  of  allowing  the  task  state  to  be 
conveniently  extended  from  within  MultiScheme,  but  it  must  still  be 
manipulated  exjjlicitly.  This  mechanism  is  not,  in  fact,  used  in  the 
current  implementation. 

3,  The  trap  mechanism  described  for  fluid-let  is  cjuite  general.  By 
placing  different  flags  (actually  a  procedure)  into  tlie  trap  object  it 
is  possible  to  specify  an  arbitrary  mechanism  for  resolving  variable 
references.  One  other  way  to  use  this  power  is  to  specify  that  refer¬ 
ences  to  a  particular  lexical  variable  are  to  be  resolved  into  references 
to  the  task  state  object,  as  implemented  in  either  of  the  earlier  two 
solutions.  This  permits  the  structuring  of  the  name  space  via  lexical 
scoping  to  be  utilized  without  either  fragmenting  the  task  state  or 
distributing  through  the  code  the  knowledge  of  which  variables  are 
task  state  variables. 

Providing  a  convenient  syntax  and  interface  to  these  mechanisms  is  an 
int('resting  area  for  further  investigation.  The  mechanisms  themselves  are 
quite  powerful  and  have  easily  satisfied  all  of  the  current  demands  for  task 
state.  Unfortunately  they  are  rather  hard  to  use  and  the  difficulties  that 
have  arisen  in  this  area  so  far  do  not  need  to  utilize  their  full  capabilities.  As 
more  complex  ai)plication.s  are  built,  these  mechanisms  will  almost  certainly 
l)e  employed  to  help  control  the  modularity  of  the  program  design. 


3.4  Introducing  the  Task 

Tlu'  history  of  MultiScheme  includes  a  long  series  of  experiments  with  the 
notion  of  a  task.  Stpele[54j  introduced  the  notion  of  a  continuation  (which 
originated  in  the  area  of  denotational  semantics)  as  a  means  of  analyzing 
Scheme  i)rograms  in  the  process  of  compilation.  This  “continuation  passing 
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style"  of  programming  proved  sufficiently  interesting  and  powerful  that  the 
notion  of  using  continuations  as  part  of  ordinary  programming  practice 
was  expanded  and  demonstrated  in  a  series  of  papers  by  Friedman  and 
others  at  Indiana  University[19, 22,29].  The  use  of  continuations  as  first- 
class  data  oljjects  was  added  to  the  language,  and  they  were  hailed  as  a 
rej)lacenient  for  the  standard  catch  and  throw  constructs  of  earlier  Lisp 
systems.  The.se  continuation  objects,  since  they  encapsulate  the  interesting 
state  of  the  system,  were  the  first  candidates  for  tasks. 

Using  them  it  is  easy  to  construct  a  multiprocessing  system,  sharing  a 
single  processor  among  a  number  of  “active”  continuations.  For  example, 
the  concept  of  an  engine  introduced  by  Haynes  and  Friedman[30]  is  one  way 
of  capturing  this  notion.  At  this  time  a  continuation  was  purely  a  control 
object.  It  was  noticed  that  a  continuation  is  completely  interchangeable 
with  a  procedure  making  it  is  easy  to  extend  the  continuation  seen  by  a 
programmer  to  include  other  pieces  of  state.  As  described  in  Section  D.5, 
the  MIT  Scheme  system  includes  the  dynamic  state  information  as  part  of 
the  continuation  objects  manipulated  by  programs.  Again,  these  extended 
continuation  objects  are  good  candidates  for  the  notion  of  a  task. 

In  multiprocessing  systems  there  is,  however,  a  notion  that  is  not  explic¬ 
itly  visible.  Not  all  continuations  are  treated  in  the  same  way.  Some  of  them 
are  candidates  for  e.xecution  by  the  system  and  others  are  merely  objects 
created  and  manipulated  by  programs.  The  notion  of  a  task  refers  to  these 
former  kinds  of  continuations.  A  continuation  embodies  work  passively, 
the  same  way  a  procedure  does  (in  fact,  some  Scheme  systems  implement 
continuations  as  procedures).  A  task,  on  the  other  hand,  embodies  work 
which  for  one  reason  or  another  should  actually  be  undertaken.  A  task  is 
active  if  a  processor  is  actually  working  on  it.  It  is  suspended  if  it  could  use 
a  processor  if  one  were  available  but  no  processor  is  actually  working  on  it. 
The  system  maintains  a  data  structure  that  contains  all  of  the  suspended 
tasks  so  that  when  the  processor  chooses  to  cease  executing  the  current 
(active)  task  it  can  choose  a  new  task  to  execute.  With  the  addition  of 
jilaceholder  objects  and  an  event  driven  system,  there  is  a  third  status  for 
tasks.  They  can  be  inactive  if  .some  event  must  occur  before  it  can  be  active 
again  typic.ally  the  event  will  be  the  arrival  of  a  value  for  a  ptirticular 
placeholder. 

The  iK'Xt  stage  in  the  evolution  of  a  MultiScheme  task  was  motivated 
by  the  desire  to  ni:d<e  the  system  demand  driven.  With  this  goal  comes  the 
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lift'd  ft)  i;ailiafi‘  l■tlli^■^'r  iiiin<‘f<'ssar\‘  task-',  and  thi>  in  tiiin  make.''  rlic  task 
i;;>r;iift  hoiii  tlia  CDUtiiiuation  ohji-ft.  Tin'  t'outinuat ion  ('luliodics  work 
it'  ijc  pt'i {oniit'tl.  But  the  tiisk  t'mhodifs  lioth  work  to  lx-  pcrforiui'd  i  ;t 
t  i.’itininition  )  .'inil  tlio  n  ii.<un  wliy  tlic  work  shouUl  Ix'  pcrforiut'd.  Shonld 
tin--  rt'iison  1  .'I'i'i .'iia'  in'/alid.  tin'll  tin*  titsk  ciai  In'  rl■Ino^■<'d  from  consitlfriit ion 
!• 'I  p'idi't'>Mii  rt'soiii'ft's,  i.t'..  It  ciin  In'  f;,;irl)ant'  collartt'd.  Tlic  t,'isk  is  ;t  way 
to  rcpri'si'ut  a  I'ont iniuit ion  that  is  rt'ijm'stiiin  processor  resources.  ;ind  the 
'ft  ot  at'ti\f  iiin!  ''U>’p('ntied  ta^ks  lire  wlnit  constitute  the  ilemiuidine  iixeiits 
of  the  (It'imiiuhi inven  systt'in, 

Mitintt-iininy  knowledge  about  reasons  for  perforitiing  work  and  scln'dnl- 
mx  based  on  these  reasons  is  the  core  of  much  work  in  iirtiflcial  intellij^ence. 
C'lciiriy  it  is  not  yet  within  the  state  of  the  art  to  use  tlie,se  techniques  for 
stdieduliusi;  and  garbage  collecting  tasks  at  the  lowest  level  of  a  program¬ 
ming  s}'-tem.  Insteail.  ;i  vt'ry  simple  operational  definition  can  be  (and  was) 
Us<'d.  The  reason  for  performing  a  ta.sk  was  not.  in  fact,  exjilicitly  spcc- 
itied.  In.'teiul.  the  stiunhird  Lisp  garbage  collector  was  used  as  a  model. 
The  Scheme  system  has  .some  de.signated  objects  (the  rC'Ot  of  the  garbage 
collection)  that  are  considered  to  lie  vital  to  the  existt'iice  of  the  system. 
I  hese  must  be  iireservetl  at  till  costs  ;md  are  their  own  reason  for  existence. 
But  they  ill  turn  depi'ini  upon  tlie  existence  of  (or  point  at  or  contain) 
other  objects.  The  reason  for  existenct*  of  these  other  objects  is  the  fact 
tln'it  the  vit.'d  objects  dept'nd  upon  them.  BtU  these  in  turn  depend  upon 
yet  other  objects,  and  so  forth.  Thus  ('very  oliject  that  is  preserved  tifter 
<1  g.'irbage  collection  remains  because  it  is  either  a  vital  object,  or  there  is 
a  dependency  chain  from  a  vital  object  to  the  oliject.  The  same  is  true  of 
tasks  a  fa.sk  !''’s  a  vtilid  rc'ason  to  exist  if  it  is  eitlu'r  one  of  the  vit.d 
objects  or  tlic're  is  sonu'  p.ith  from  the  vital  objects  to  reach  the  task. 

But  in  ;i  very  real  sense  this  begs  the  qut'stion.  What  are  the  vit:d 
objects  in  the  system?  In  MIT  Scheme  tlu're  are  a  large  number  of  them, 
Imt  only  two  typically  lead  to  retaining  tasks.  The  first  is  the  global  eu- 
\  iroum«'nt  sinci'  the  values  of  varialrles  in  this  environment  can  always  be 
refcif'nccd’^.  Introducing  a  global  name  for  a  task  is  tantamount  to  declar¬ 
ing  that  the  task  must  always  b(’  iivtdlable,  ami  h('nce  it  has  a  permam'Ut 


'Ill'  access  sjuTial  fnnii  in  MIT  Srln'ini'  can  ix*  lood  in  any  li'Xiral  envinjnnu'nt 
rif.r>nct'  vanaMi-s  in  (he  glohal  cnvircjiinu'iil .  I'lic  glohal  environment  is  the  only 
environnieiit  for  which  this  is  true,  since  it  is  the  only  eiivironinent  tliat  can  itself  lie 
ri  f.  rence.l  wiliioiit  i  he  use  of  a  natne. 


V. 


3.4.  INTRODUCING  THE  TASK 


79 


reason  for  existence.  Similarly,  storing  a  task  in  a  data  structure  or  vari¬ 
able  that  can  be  located  by  a  chain  of  ordinary  accessors  from  the  global 
environment  gives  the  task  a  reason  to  exist. 

The  second  object  of  interest  is  the  task  executing  at  the  time  of  the 
garbage  collection.  Since  the  garbage  collection  primitive  operation  is  in¬ 
voked  as  any  other  primitive  function  it  runs  as  part  of  the  task  that  calls 
it.  But  the  fact  that  a  task  was  active  at  the  time  of  a  garbage  collection  is 
hardly  a  good  reason  to  maintain  the  task.  As  a  result,  the  interrupt  han¬ 
dler  that  initiates  a  garbage  collection  (see  Section  3.1.1)  guarantees  that 
all  processors  switch  to  a  task  that  exists  only  for  the  purpose  of  garbage 
collection. 

The  addition  of  placeholders  to  Scheme  puts  a  new  complexion  on  this 
problem.  If  we  use  tasks  primarily  as  part  of  the  future  macro  or  a  similar 
construction,  then  we  always  create  a  placeholder  for  the  value  of  a  task 
at  the  same  time  the  task  is  created.  In  a  very  real  sense,  the  purpose 
of  the  task  is  to  calculate  a  value  for  that  placeholder.  This  is  reinforced 
by  the  fact  that  in  a  parallel-  or  multi-processing  system  the  placeholder 
propagates  through  the  system  even  while  the  value  it  represents  is  being 
calculated.  Thus  we  need  to  represent  the  fact  that  a  task’s  purpose  is  the 
calculation  of  the  value  for  a  specific  placeholder.  This  is  easily  done  by 
making  the  placeholder  point  to  the  task  calculating  its  value.  In  this  way 
if  the  garbage  collector  maintains  any  references  to  the  placeholder  it  will 
also  retain  the  task  itself. 

The  final  step  in  the  evolution  of  the  task  comes  from  the  decision  to 
firmly  link  the  notion  of  task  to  placeholders.  Associating  every  task  in 
the  system  with  a  placeholder  and  making  that  placeholder  the  “goal”  for 
the  task  provides  a  simple  linkage  between  tasks  and  placeholders  that 
makes  it  considerably  easier  to  grasp  the  notion  of  task.  Tasks  exist  for  the 
I)urpose  of  computing  some  value,  and  the  placeholder  associated  with  the 
task  allows  this  value  to  propagate  through  the  system  even  while  the  task 
is  doing  the  calculation 

This  simplification  is  nice,  but  it  is  important  to  keep  in  mind  that 
a  task  may  do  other  things  in  the  process  of  calculating  the  value  of  its 
a-,s()(  iated  placeholder.  There  is  an  important  analogy  between  tasks  with 
a  p'li  ])(jse>  and  procedures.  We  may  think  of  a  procedure  as  existing  for  the 
of  coniputing  some  value.  Yet  some  very  useful  procedures  loop 
:  '  and  jJioduce  no  value.  Similarly,  a  task  may  include  an  infinite  loop 


CHA P TEPx  3  PARALLEL  PR O CESSISG  EX TEXSIOXS 


1 

I 

I 


Si) 


and  la'iice  its  associatt'd  plaocholdor  may  iiovt'r  icccivo  a  vahu'.  Siiuilaily. 
a  pix)aedur<'  may  (.Udihcratcly  sid<*-cfFoot  a  variable  or  data  struct  m  e  or 
invoke  a  eotitinuatioii  thereby  producing  a  result  in  a  place  otlier  tiian 
the  one  indicated  by  its  ostensible  purpose.  Likewise,  a  task  may  ext'cute 
cocle  that  contains  an  explicit  call  to  determine!  and  thereby  contribute 
a  value  to  a  placeholder  other  than  the  one  it  is  apparently  iuttnided  to 
compute.  The  association  between  the  task  and  the  placeholder  shouhl  be 
thought  of  as  providing  a  form  of  ordinary  behavior  chosen  to  support  the 
expected  common  patt  n  of  usage  rather  than  an  inherent  and  unshakeable 
condition. 

To  summarize  this  model,  here  is  a  recap  of  the  three  main  MultiScheme 
objects  relevant  to  (potentially)  simultaneous  computations. 

Placeholders 

A  placeholder  represents  a  value  that  is  not  yet  computed.  Place¬ 
holders  associated  with  a  task  (or  a  set  of  tasks)  provide  the  primary 
“computational  motive  force”  for  those  tasks.  There  are  typically 
no  references  to  tasks  except  through  the  placeholders  for  the  values 
they  are  calculating.  Touching  a  placeholder  may  cause  a  task  to  be 
generated,  providing  a  lazy  evaluation  mechanism. 

Continuations 

A  continuation  represents  a  control  state,  a  set  of  fluid  variable  bind¬ 
ings,  and  a  corresponding  point  in  the  system-state-space®  (see  .4p- 
pendix  D).  These  three  parts  can  be  restored  independently,  or  they 
can  be  restored  jointly  by  applying  the  continuation  to  an  argument. 
This  corresponds  to  transfer  of  control  (goto)  in  other  languages. 
There  is  a  special  continuation,  used  when  a  task  is  generated,  that 
corresponds  to  task  termination.  It  stores  the  value  supplied  to  the 
continuation  as  the  value  of  the  placeholder  associated  with  the  task 
that  is  terminating. 

Tasks 

.4  task  represents  a  line  of  computation  that  is  expected  to  ultimately 
lead  to  the  computation  of  a  value  for  a  particular  placeholder.  ^V'lien 

'( 'out  iiiuat ions  also  contain  otlier  information  relevant  to  tlie  control  slate  of  the  sys- 
ti'iii,  such  as  the  inlrTrupt  iiia.sk  and  compiler  register  .set.  The  fact  tlial  this  is  not  encoiied 
m  tile  control  state  itself  is  an  implementation  detail. 
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a  task  is  created  the  placeholder  that  is  its  goal  must  be  specified,  and 
that  placeholder  forms  the  primary  source  of  “computational  motive 
force"  that  will  keep  the  task  running.  Typically,  if  the  placeholder 
is  no  longer  needed  the  task  will  be  removed  from  the  system  (along 
with  the  placeholder),  although  it  is  possible  to  retain  a  task  explicitly 
if  necessary.  Tasks  are  created  with  the  special  “task  termination” 
continuation  which  will  cause  the  value  supplied  to  the  continuation 
to  be  placed  in  the  task’s  placeholder  and  then  terminate  the  task. 


3.5  Demand-Driven  Computation 

One  of  the  important  design  factors  in  MultiScheme  is  that  computation 
lie  demand  driven  —  not  in  the  sense  of  using  “call  by  need”  parameters 
or  a  "lazy  evaluator,”  but  rather  in  the  sense  that  a  computation  continues 
to  run  only  as  long  as  it  is  serving  a  useful  purpose.  The  notion  of  a  task, 
introduced  in  Section  3.4,  captures  the  essence  of  this  drive  by  providing 
an  explicit  goal  for  each  computation  in  the  system.  One  way  of  stating 
this  demand-driven  computation  mode  is  by  saying 

A  computation  continues  only  as  long  as  the  value  it  is  com¬ 
puting  is  still  needed. 

By  reversing  the  statement,  it  can  provide  the  answer  to  a  problem 
raised  when  converting  a  standard  sequential  Scheme  program  into  a  par¬ 
allel  version. 

As  long  as  the  value  of  a  computation  is  still  needed  that  com¬ 
putation  must  continue. 

The  parallel  between  memory  allocation  and  task  allocation  is  very 
strong.  .Tust  as  Lisp  allows  memory  to  be  allocated  and  invisibly  handles 
the  release  of  that  memory  when  it  is  no  longer  needed  by  the  computation 
at  hand,  MultiScheme  allow's  tasks  to  be  created  and  will  remove  them 
wlien  they  are  no  longer  needed.  Thus  the  need  to  abort  a  task  is  mini¬ 
mized.  just  as  the  need  to  deallocate  memory  is  minimized.  Furthermore 
the  t('chnique  used  in  Ijoth  ca.ses  to  force  the  deallocation  is  the  same.  To 
release  memory  the  programmer  must  find  all  objects  that  reference  the 
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uu'iiun  y  being  released  and  guarantee  that  they  no  longer  reference  the  ob¬ 
ject  in  question  (typically  this  means  storing  ’  ()  or  some  other  non-pointer 
as  the  value  of  a  variable  that  is  normally  the  only  entry  point  to  a  data 
structure).  Similarly,  in  MultiScheme  the  programmer  must  guarantee  that 
the  goals  of  all  the  unneeded  computations  have  been  provided  a  value  by 
the  explicit  use  of  determine ! 

This  item  of  jdiilosojjliy  becomes  most  obvious  when  considering  the  use 
t)f  call-with-current-continuation  (creation  of  a  continuation  object) 
tind  throw  (the  use  of  such  an  object).  In  a  single  proce.ssor  system  there  are 
a  immlier  of  wa}-s  to  understand  this  interaction,  and  a  number  of  st}'les  for 
their  use.  In  Scheme,  contiriuations  are  merely  a  convenient  way  of  writing 
certain  procedures.  The  decision  to  explicitly  write  out  a  procedure  or  to 
make  use  of  a  continuation  is  purely  one  of  user  convenience  and  should 
imply  no  deep  consequences.  Based  on  this  line  of  reasoning,  continuations 
in  MultiScheme  are  treated  exactly  as  any  other  procedure  object.  They 
have  the  effect  of  changing  the  implicit  continuation  being  used  by  the  task 
that  calls  them,  but  the  task  aird  its  goal  remain  intact. 


(define  (f  y) 

(call-with-current-continuation 
(lambda  (send-answer) 

(define  (transform  fn) 

(let  ((fn-of-y  (fn  y))) 

(if  (=  0  fn-of-y) 

(send-answer  ’no-good) 
(hairy-computation  fn-of-y)))) 
(+  (transform  sin)  (transform  cos))))) 


Figure  3.4:  Multi-level  Exit  using  Call-With-Current-Continuation 


This  decision  is  not  without  ramifications  and  is  not  taken  lightly.  By 
adopting  a  demand-driven  model  and  the  notion  of  goal-oriented  tasks  it 

‘”For  more  drcistic  cases  it  is  possible  to  use  within-task  to  force  a  teisk  to  halt,  but 
this  can  leave  the  system  in  an  inconsistent  state  if  not  done  with  care. 
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is  no  longer  true  that  (touch  (future  <exp>))  is  equivalent  to  <exp>  I 

in  functional  programs.  Consider  the  program  of  Figure  3.4.  This  is  a  | 

fairly  common  use  of  call-with-current-continuation  where  it  is  used  j 

to  provide  a  way  to  return  a  result  from  a  nested  procedure  call.  This 

use  is  similar  to  the  way  catch  is  used  in  CommonLisp.  If  transform  j 

dt'tects  a  problem  condition  it  immediately  returns  an  answer  of  NO-GOOD 
ro  the  program  that  called  F  and  neither  the  hairy  computation  nor  the 
addition  operation  is  performed.  Thus,  for  example,  (future  (f  0))  will 
viltimately  return  an  answer  of  NO-GOOD. 

If  we  wish  to  introduce  parallelism  into  this  program,  one  obvious  way 
to  do  it  would  be  to  perform  the  calls  to  transform  simultaneously.  Thus 
we  would  change  the  final  line  of  the  program  to 

(+  (future  (transform  sin)) 

(future  (transform  cos))) 

The  program  now  results  in  an  error  since  (future  (transform  sin) ) 
creates  a  new  task  whose  goal  is  the  computation  of  the  transform.  This 
task  then  uses  the  send-answer  continuation  to  deliver  a  result  of  NO-GOOD 
as  the  result  of  the  call  to  (f  0).  Since  this  is  the  end  of  a  task  (the  one 
created  by  (future  (f  0)),  the  answer  is  stored  in  the  goal  of  the  current 
task  —  and  this  is  then  added  to  the  other  transform,  resulting  in  the  error. 


(define  (alternate-answer  body) 

(define  final-result 

(delay  (body  (lambda  (early-answer) 

(determine!  final-result  early-answer) 
(kill-task) ) ) ) ) 

(touch  final-result)) 


Figure  3.5:  A  Replacement  for  Call-With-Current-Continuation 


What  is  the  basis  of  thi.s  problem?  In  terms  of  the  demand-driven 
computing  model  the  difficulty  here  is  that  one  particular  task  has  the  goal 
of  computing  a  value  for  (f  0).  In  order  to  achieve  thi.s  goal,  additional 
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tasks  ;ire  crt'afccl  and  the  goals  for  these  tasks  would  ordinarily  be  combined 
by  tlie  original  task  to  form  the  value  of  its  goal.  The  use  of  a  continuation 
here  is  intended  to  allow  any  task  to  provide  a  value  for  this  goal. 

This  can  be  easily  expressed  by  replacing  the  call  to  call-with-cur- 
rent-continuation  with  a  slightly  different  procedure,  alternate-answer 
shown  in  Figure  3.5.  This  works  by  creating  a  new  task  to  execute  the  body 
of  code  that,  in  the  normal  case,  is  executed  by  the  originating  task.  This 
new  task  computes  as  usual,  but  the  procedure  that  replaces  the  continu¬ 
ation  in  the  original  version  explicitly  determines  the  placeholder  for  the 
generated  task.  Thus,  regardless  of  which  task  tries  to  use  this  continua¬ 
tion,  it  is  the  original  placeholder  that  receives  the  value.  In  this  example, 
the  task  that  generates  the  answer  then  “commits  suicide,”  leaving  its  own 
placeholder  without  a  value'’  on  the  assumption  that  there  can  be  no  need 
for  the  task’s  value  if  the  alternate  answer  is  being  returned.  If  the  task 
should  continue  computing  then  the  call  to  kill-task  can  be  omitted. 

This  solution  also  demonstrates  one  important  consequence  of  the  de¬ 
mand-driven  approach.  Using  continuations  for  multi-level  exit  on  a  parallel 
processor  leaves  open  the  question  of  how  to  kill  other  tasks  that  were 
spawned  after  the  call-with-current-continuation  was  executed  by  the 
original  task.  This  “spawning  tree”  oriented  approach  to  programming  can 
be  completely  avoided  in  MultiScheme.  The  problem  is  non-existent:  those 
tasks  that  continue  to  compute  after  a  value  is  available  for  the  placeholder 
created  by  the  call-with-current-continuation  are  precisely  those  that 
are  attempting  to  compute  values  needed  by  the  system  even  when  that 
value  is  known.  To  have  killed  these  tasks  merely  because  of  the  spawning 
relation  would  lead  to  deadlock  later  - —  a  problem  avoided  by  allowing  the 
garbage  collector’s  traversal  of  active  memory  to  detect  and  remove  tasks 
that  are  unreferenceable. 

3.6  Summary 

Extending  MIT  Scheme  for  use  on  a  parallel  processing  system  required 
four  major  areas  to  be  explored:  processor  coordination,  data  storage  for 

’’The  procedure  kill-task  isn’t  specified  here,  so  it  can  arrange  to  place  an  error  value 
in  the  placeholder  if  desired.  Otherwise,  the  procedure  next  discussed  in  Section  4.2.2 
would  be  a  reasonable  choice. 
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tasks,  dyiiainic  state  and  exception  handling,  and  the  distinction  betwcf'ii 
tasks  and  continuations. 

Section  3.1  introduced  the  mechanisms  used  to  coordinate  the  processors 
in  the  system:  synchronizers  which  act  as  a  “starting  gate”  to  guarantee 
that  all  processors  are  at  a  given  part  of  the  computation,  and  global - 
interrupt  to  get  the  attention  of  all  processors  and  demand  their  cooper¬ 
ation  in  initiating  some  event.  These  were  used  to  support  three  important 
system  operations;  initiating  garbage  collection,  pausing  the  system  for 
observation,  and  reacting  to  user  interrupt  requests. 

The  separation  of  the  name  space  in  a  way  which  enables  tasks  to  access 
information  that  they  may  not  necessarily  share  with  others  is  discussed 
in  Section  3.3.  The  problem  is  divided  into  two  different  jobs.  The  fluid 
variables  of  MIT  Scheme  are  extended  into  the  parallel  processing  domain 
to  provide  one  form  of  support.  In  addition,  three  different  techniques  (see 
page  75)  for  providing  totally  private  task  information  were  discussed. 

Finally,  the  distinction  between  a  continuation  and  a  task  in  Multi- 
Scheme  was  introduced  in  Section  3.4.  The  motivation  for  this  distinction 
and  the  important  notion  of  demand-driven  computation  was  introduced. 
The  use  of  task  goals  to  support  the  garbage  collection  of  useless  tasks 
created  for  speculative  computation  was  described  in  Section  3.5. 
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Chapter  4 

Implementing  the  Scheduler 


The  MultiScheme  scheduler  provides  a  convenient  interface,  in  the  form  of 
a  package  of  procedures,  between  MultiScheme  programs  and  the  under¬ 
lying  virtual  machine.  Some  of  the  procedures  are  invoked  by  programs 
written  in  MultiScheme  while  others  axe  invoked  as  part  of  the  trap  or 
interrupt  handling  of  the  virtual  machine.  The  scheduler  is  itself  written 
in  MultiScheme  and  is  relatively  small  (20  pages  of  code  including  utility 
routines).  This  has  proven  to  be  an  important  factor  in  the  development 
of  MultiScheme,  providing  a  localized  and  flexible  base  for  a  number  of 
experiments  with  the  nature  of  event-driven  computing. 

This  chapter  serves  four  purposes.  First,  it  constitutes  a  “proof  by 
example”  that  the  MultiScheme  laxiguage  as  described  in  Chapters  2  and  3 
is  a  powerful  systems  programming  language  for  a  parallel  processor.  It 
also  serves  as  the  first  extended  example  of  programming  in  this  language. 
Third,  it  provides  an  English  description  of  a  number  of  the  constructs  used 
in  the  implementation  and  thus  acts  as  a  form  of  documentation  for  people 
dealing  with  the  actual  program.  Finally,  the  implementation  of  disjoin 
in  Section  4.3.2  and  Figure  4.9  fulfills  the  promise  made  in  Section  2.5  of 
demonstrating  an  event-driven  mechanism  for  implementing  the  aiinb  and 
fair-merge  operations. 

This  chapter  discusses  each  of  the  major  operations  supported  by  the 
scheduler:  task  creation  (Section  4.3),  task  suspension  and  task  switch 
(Section  4.4),  storing  a  value  into  a  placeholder  (Section  4.5),  and  transition 
from  parallel  processing  to  single  task  execution  (Section  4.6).  The  rough 
outline  of  the  scheduler  (the  services  it  supports  and  the  inter-relationship 
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betwet'ii  these  services)  has  prov-eti  quite  robust  over  time.  Even  as  tli(' 
system  grew  to  support  more  kinds  of  event  driven  computation,  the  c()r(' 
of  the  sch('dul('r  ;is  described  here  has  remained  almost  constant.  The 
scheduler  was  originally  intended  to  be,  and  remains,  a  highly  flexible  laxly 
of  c(xie.  The  scheduler  described  here  is  the  “standard"  scheduler  as  it 
currt'utly  exists.  As  new  applications  are  developed,  driving  the  system 
towtird  new  modes  of  computation,  the  data  structures  of  the  scheduler  are 
modified  to  accommodate  the  new  requirements.  Users  are  encouraged  to 
examine  and  understand  the  scheduler,  and  feel  free  to  modify  it  for  their 
own  needs.  Naturally,  such  modifications  must  be  undertaken  with  a  good 
deal  of  care.  But  these  modifications  have  proven  useful  in  the  past  and 
have  in  some  cases  been  formalized  and  added  to  the  standard  MultiScheme 
scheduler. 

By  its  very  nature,  the  discussion  in  this  chapter  is  more  closely  focused 
on  implementation  details  than  are  the  earlier  chapters.  The  presentation 
is  roughly  bottom  up,  describing  the  data  structures  in  Section  4.1,  general 
utility  routines  in  Section  4.2,  and  then  the  user-visible  routines.  In  order  to 
avoid  an  overwhelming  amount  of  detail  the  examples  included  in  this  chap¬ 
ter  are  simplified  versions  of  the  actual  procedures  in  the  scheduler.  These 
simplified  versions  present  the  important  core  of  each  procedure,  and  should 
be  considered  more  closely  related  to  pseudo-code  than  to  fully  worked  out 
implementations.  In  many  cases  the  versions  presented  here  will  not  work 
correctly  in  the  actual  implementation  of  MultiScheme.  This  comes  from  a 
variety  of  reasons,  including  race  conditions  and  name  changes  introduced 
to  be  more  consistent  with  the  terminology  of  this  document.  Readers  in¬ 
terested  in  the  complete  versions  of  these  procedures  should  contact  the 
author  for  a  current  version  of  the  scheduler  code. 

4.1  Scheduler  Data  Structures 

Much  of  the  work  of  the  scheduler  procedures  revolves  around  the  correct 
maintenance  of  the  data  structures  that  implement  placeholders  (see  Sec¬ 
tion  2.1),  tasks  (see  Section  3.4)*,  and  a  queue  of  tasks  that  are  ready  to 

'At  this  time,  the  task  and  placeholder  data  structures  are  actually  implemented  a,s  a 
single  Scheme  object.  This  works,  but  is  both  a  conceptually  poor  idea  and  implementa- 
tionally  clumsy.  The  work  of  splitting  the  two  apart  has  been  planned  and  scheduled,  but 
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run. 

For  this  description,  all  access  to  data  structures  is  assumed  to  be 
through  mutators  and  selectors  for  each  part  of  the  structure.  Thus,  corre¬ 
sponding  to  the  goal  slot  of  a  ta.sk  data  structure  there  are  two  procedures: 
task. goal  returns  the  goal  of  a  given  task  and  set-task. goal!  stores  a 
new  goal  into  the  task  data  structure. 

4.1.1  Placeholders 

Placeholders  are  the  primary  vehicle  connecting  the  scheduler  (and  hence 
programs  written  in  MultiScheme)  with  the  underlying  support  for  parallel 
processing.  Placeholders  are  created  by  a  scheduler  procedure,  normally  as 
part  of  the  task  creation  process  (see  Section  4.3).  Supplying  a  value  for  a 
placeholder  (through  determine!  and  mutably-determine ! )  is  also  sup¬ 
ported  by  scheduler  procedures  (see  Section  4.5).  Detection  of  placeholders 
and  automatically  forcing  them  is  built  into  the  primitive  operations  and 
the  underlying  machine  itself,  as  described  in  Section  4.4. 


Name 

Notes 

Determined? 

Lock 

Value  or  W^liting  Queue 
Motivated  Teisk 

^ee  text 

See  text  for  complete  description 

Figure  4.1:  Placeholder  Data  Structure 


The  placeholder  data  structure  is  shown  in  Figure  4.1.  Each  of  the  fields 
is  described  below. 

Determined? 

A  tri-state  flag  that  indicates  whether  the  placeholder:  (a)  has  no 
value  yet;  (b)  has  an  immutable  value;  or  (c)  has  a  mutable  value. 
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This  flag  is  used  by  the  underlying  machine  to  test  whether  a  touch 
of  this  placeholder  should  trap  into  the  scheduler  (as  discussed  in 
Section  4.4)  or  extract  the  current  value  and  continue. 


Lock 

A  standard  mutual  exclusion  lock  used  to  indicate  that  the  place¬ 
holder  is  currently  being  modified  by  MultiScheme  code.  The  under¬ 
lying  machine  uses  this  lock  to  implement  the  primitive  mutual  ex¬ 
clusion  procedures  lock-placeholder!  and  unlock-placeholder!. 
In  addition,  when  a  placeholder  is  locked  the  underlying  machine 
will  not  convert  a  placeholder  into  its  value  during  garbage  collec¬ 
tion  or  variable  look-up.  Under  ordinary  circumstances  this  conver¬ 
sion  (called  splicing)  is  an  important  optimization.  The  lock  permits 
MultiScheme  code  that  explicitly  manipulates  placeholders  (typically 
within  the  scheduler)  to  suppress  this  splicing.  This  assures  that  a 
placeholder  will  not  suddenly  transform  into  another  object  while  the 
data  structure  itself  is  being  examined. 

Value 

Stores  the  value  of  the  placeholder  if  it  is  either  mutably  or  immutably 
determined. 

Waiting  queue 

A  queue  of  tasks  currently  waiting  for  this  placeholder’s  value  to  be 
determined.  This  queue  is  built  using  weak  cons  cells,  as  described 
earlier  in  Section  2.2.2,  since  membership  in  this  queue  does  not  con¬ 
stitute  a  reason  for  the  trisk  to  continue  computing.  The  underlying 
machine  does  nut  reference  this  information  (it  is  handled  only  by 
MultiScheme  code,  typically  within  the  scheduler).  Since  this  queue 
must  be  empty  when  the  placeholder  already  has  a  value,  the  imple¬ 
mentation  overlaps  the  storage  space  for  these  last  t\vo  items.  There 
is  no  inherent  reason  why  this  structure  is  a  queue  (rather  than,  say, 
a  stack)  since  all  of  the  items  are  released  for  execution  at  the  same 
time. 

Motivated  task 

The  task  that  has  the  computation  of  a  value  for  this  placeholder  as 
its  goal.  As  with  any  item  other  than  a  weak  cons  cell,  the  garbage 
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collector  does  trace  through  this  link.  Thus  it  serves  to  retain  the 
task  that  is  computing  the  value  of  this  placeholder  as  long  as  the 
placeholder  itself  is  needed.^ 

4.1.2  Tasks 

The  task  data  structure  contains  a  variety  of  information,  but  is  not  directly 
referenced  by  the  underlying  machine.  Tasks  represent  work  that  has  been 
recjuested  to  be  performed,  and  they  are  the  objects  that  the  scheduler  has 
the  underlying  machine  store  on  its  work  distribution  queue  (see  below). 

In  order  to  support  garbage  collection  of  no  longer  useful  tasks,  the  root 
used  by  the  garbage  collection  algorithm  contains  a  particular  set  of  tasks 
whose  continued  existence  is  required  by  the  user  interface  to  MultiScheme 
(those  that  can  be  reached  directly  using  the  ordinary  keyboard  interrupt 
characters).  Other  tasks  are  retained  only  if  they  can  be  reached  either 
from  this  initial  set  of  tasks  (because  one  of  these  initial  tasks  is  waiting 
for  (see  below)  the  value  of  a  placeholder  and  that  placeholder’s  motivated 
task  references  another  task)  or  from  the  global  environment. 


Name 

Notes 

Goal 

Placeholder  associated  with  this  task 

Lock 

Code 

Work  to  be  performed  when  task  is  next  run 

Status 

See  text  for  details 

Original  Code 

For  debugging  purposes 

Task-Private  Data 

See  Section  3.3.5 

Waiting  For 

Placeholder(s)  for  which  this  task  is  waiting 

Wake-up  Value 

See  text  for  details 

See  text  for  complete  description 

Figure  4.2:  Task  Data  Structure 


The  task  data  structure  is  shown  in  Figure  4.2. 

"Tlie  motivated  task  could  he  extended  to  a  list  of  tasks,  but  the  code  shown  in  this 
hapter  does  not  support  this  option. 
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Goal 

The  placeholder  that  is  the  goal  for  this  task.  When  a  task  is  ac¬ 
tively  computing,  this  placeholder  is  known  as  the  current  placeholder 
for  the  processor  doing  the  coniputation.  When  a  task  executes  the 
termination  continuation  (see  Section  4.3.3)  it  stores  the  computed 
value  into  this  placeholder. 

Lock 

A  standard  lock  to  serialize  access  to  the  task  description. 

Code 

The  code  to  run  in  order  to  re-activate  this  task.  If  the  object  stored 
here  is  not  applicable  (i.e.  neither  a  procedure  nor  a  continuation) 
then  either  the  task  is  already  active  or  for  some  reason  it  cannot  be 
reactivated  (it  may  have  finished  computing  and  not  yet  been  garbage 
collected,  for  example). 

Status 

The  current  state  of  this  task.  This  is  one  of: 

created  Task  is  newly  created 
delayed  See  delay-policy,  Section  4.3.2 
determined  Task  is  finished 

disjoin  Waiting  for  the  first  of  several  placeholders 
paused  Stopped  by  pause-everything 
rumiable  Available  for  execution 
running  .Actually  in  possession  of  a  processor 
waiting  Waiting  for  a  specific  placeholder 
within-task  Running,  but  see  Section  3.1.4 


Original  code 

For  debugging  purposes  this  contains  the  expression  that  the  ta.^k  a  a 
created  to  evaluate. 


Task-private  data 

See  Section  3.3.5. 
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are  the  responsibility  of  the  MultiScheme  code,  not  the  primitive  itself. 
Thus,  it  is  possible  for  one  task  to  drain  the  work  queue  while  other  tasks 
are  still  active  and  creating  new  entries  on  the  queue.  The  results  will  be 
consistent  (a  task  that  is  available  for  work  will  be  removed  either  by  a  call 
to  drain-work-queue  or  a  call  to  get-work  but  not  both),  although  it  may 
not  represent  an  instantaneous  snapshot  of  the  internal  data  structures. 


4.2  Overall  Concepts  and  Utility  Routines 

The  scheduler  is  organized  around  the  data  structures  described  in  Sec¬ 
tion  4.1  and  two  additional  notions.  The  first,  described  in  Section  4.2.1, 
is  atomicity  and  critical  sections  of  code.  These  are  supported  through  a 
system  of  priority  interrupts  (within  a  single  processor)  and  a  set  of  data 
object  locks  (between  processors).  The  second  is  the  task  state  and  task 
switch  operations,  described  in  Section  4.2.2,  supported  through  the  use  of 
Scheme’s  continuations. 

4.2.1  Atomicity 

As  with  any  operating  system  scheduler,  most  of  the  routines  in  the  sched¬ 
uler  must  appear  to  occur  without  interruption.  The  fact  that  these  rou¬ 
tines  are  written  in  Scheme,  however,  does  not  permit  them  to  run  com¬ 
pletely  uninterrupted:  the  garbage  collector  cEinnot  be  suppressed  for  even 
short  intervals  without  serious  consequences.  As  a  result,  most  of  the  oper¬ 
ations  are  written  to  raise  their  own  interrupt  level  to  prohibit  any  kind  of 
interrupt  except  garbage  collection,  and  the  garbage  collection  code  guar¬ 
antees  that  any  task  that  is  running  at  a  raised  interrupt  level  will  continue 
to  run  after  the  garbage  collection.  This  notion  is  embedded  in  the  macros 
atomic  and  def  ine-atomic  which  are  used  liberally  throughout  the  sched¬ 
uler  implementation.  To  make  the  code  more  easily  understood,  however, 
these  have  been  omitted  from  the  simplified  versions  described  here. 

A  second  standard  problem,  exclusive  access  to  certain  data  structures, 
also  exists  in  the  scheduler.  As  Halstead[26]  shows,  users  can  ordinar¬ 
ily  use  placeholders  and  certain  primitive  atomic  operations  to  implement 
semaphores.  These  solutions  can  be  extended  to  include  other  standard 
multi-  and  parallel-processing  interlocks.  Unfortunately,  the  scheduler  is  in 
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Waiting  for 

If  the  task  has  status  waiting  or  disjoin  this  specifies  the  placeholder 
(or  list  of  placeholders)  for  which  it  is  waiting.  This  is  an  ordinary 
(strong)  list,  since  the  value  of  these  placeholders  is  necessary  for  this 
task  to  continue  its  own  computation.  Hence  this  task  represents 
a  reason  for  the  tasks  that  axe  computing  a  value  for  each  of  these 
placeholders  to  continue  their  computation. 

Wake-up  value 

When  the  task  is  awakened,  the  code  is  passed  this  value  as  its  ar¬ 
gument.  It  is  primarily  used  in  the  implementation  of  disjoin  (see 
sections  4.3.2  and  4.4). 

In  addition,  it  has  long  been  expected  that  some  information  might  be 
stored  here  for  use  by  user  supplied  scheduling  policies.  This  information 
could  indicate  task  priority  or  estimated  time  required  for  the  value  to  be 
computed.  This  has  not  been  implemented  since  no  applications  have  yet 
required  sophisticated  scheduling  policies. 

4.1.3  Runnable  Task  Queue 

The  subject  of  task  distribution  is  one  on  which  a  great  deal  of  work  has 
been  done,  both  theoretical  and  practical.  The  MultiScheme  scheduler  is 
built  using  three  primitive  operations  (put-work,  get-work,  and  drain- 
work-queue)  that  encapsulate  the  choice  of  task  distribution  mechanism 
implemented  by  the  underlying  machine.  The  model  employed  by  the 
scheduler  is  that  it  can  announce  that  a  task  needs  processing  resources 
using  put -work,  and  when  it  needs  more  work  to  do  it  will  retrieve  a  teisk 
using  get-work.  Because  the  shared  heap  contains  all  of  the  task,  place¬ 
holder,  stack,  and  other  structures  needed  for  a  computation,  no  guarantee 
is  made  that  a  task  will  be  run  on  the  same  processor  which  announced 
that  it  needed  resources. 

In  addition,  there  are  times  when  the  system  must  retract  work  that  has 
been  declared  to  be  available  (such  as  during  garbage  collection  initiation). 
The  primitive  operation  drain-work-queue  returns  a  standard  data  struc¬ 
ture  (composed  of  weak  cons  cells)  containing  all  items  that  were  available 
for  work  at  the  time  the  primitive  was  called.  It  leaves  the  work  ciueue  of 
the  underlying  machine  empty.  As  usual,  synchronization  and  serialization 
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large  part  responsible  for  implementing  the  semantics  for  placeholders  that 
these  solutions  exploit.  The  scheduler,  therefore,  is  built  using  two  major 
utility  routines  that  are  in  turn  based  on  spin-locks. 

The  primitive  operation  lock-placeholder!  is  used  to  implement  the 
more  complicated  of  the  two  utility  routines.  It  is  called  with  one  eirgument 
that  is  normally  a  placeholder.  Lock-placeholder  I  either  immediately 
returns  ^f  (if  its  argument  is  not  a  placeholder)  or  it  waits  until  it  is  able 
to  acquire  the  lock  which  is  part  of  the  placeholder  data  structure  and 
returns  a  value  of  #t^. 

Using  this  primitive,  the  following  scheduler  utility  routine  can  be  writ¬ 
ten: 

(define  (With-Placeholder-Locked  Placeholder  Procedure) 

(atomic 

(if  (lock-placeholder!  Placeholder) 

(let  ((result  (Procedure  #t))) 

(unlock-placeholder!  Placeholder) 
result) 

(Procedure  #f)))) 

As  can  be  seen,  With-Placeholder-Locked  runs  a  procedure  with  a 
given  object  (expected  to  be  a  placeholder)  locked.  The  procedure  receives 
an  argument  that  indicates  whether  the  object  is  in  fact  a  placeholder. 
Notice  that  when  the  object  is  not  a  placeholder  (and  hence  the  procedure 
is  given  the  argument  #f )  the  object  is  not  locked  while  the  procedure  is 
running,  since  MultiScheme  does  not  provide  any  standard  way  of  locking 
arbitrary  objects. 

A  similar  utility  routine,  With-Task-Locked  is  also  supplied.  It  locks  a 
task  (its  first  argument)  and  then  calls  a  procedure  (its  second  argument) 
indicating  whether  the  task  is  actually  runnable.  When  the  procedure 
returns  the  task  is  unlocked.  Unlike  With-Placeholder-Locked,  the  ta^k 
is  always  locked  when  the  procedure  runs  since  the  race  condition  that 
exists  for  placeholders  is  not  a  problem  with  tasks. 

primitive  is  included  for  this  purpose  only  because  of  the  complexity  of  writing  the 
correct  MultiScheme  code  to  deal  with  a  potential  race  between  one  processor  setting  the 
value  of  a  placeholder  and  another  processor  attempting  to  lock  it. 
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4.2.2  Task  Switch 

When  a  processor  changes  tasks  it  is  really  performing  into  three  separate 
operations.  The  first  operation  captures  the  current  state  of  the  task  in  a 
way  that  allows  it  to  be  restarted  later.  The  second  chooses  a  new  task 
for  execution,  and  the  third  activates  a  chosen  task.  Task  termination  (as 
described  in  Section  4.3.3)  is  nothing  more  than  performing  the  last  two 
steps  but  not  the  first.  Task  creation  (see  Section  4.3.2)  may  include  the 
first  and  third  steps  with  a  standard  choice  for  the  second. 

The  first  operation,  capturing  the  current  state  of  a  task,  is  done  using 
a  variant  on  MIT  Scheme’s  call-with-current-continuation  procedure. 
Once  a  task  is  suspended  it  will  be  resumed  only  once,  and  then  that  state 
will  later  be  suspended  and  so  forth.  Thus,  unlike  an  ordinary  continuation 
object,  the  object  that  denotes  a  suspended  t^lsk  state  need  not  be  able 
to  be  invoked  multiple  times.  In  implementation  terms,  this  means  that 
a  certain  amount  of  copying  of  continuation  stack  entries  can  be  avoided 
with  task  suspensions.  While  this  detail  is  an  important  efficiency  concern, 
it  has  not  been  called  out  in  the  code  shown  here. 

While  acquiring  a  representation  of  the  current  state  of  the  computation 
is  simple,  actually  storing  it  in  the  task  data  structure  is  not  as  straight¬ 
forward.  In  Section  3.1.4  an  important  user  operation,  within-task,  was 
introduced  (the  relevant  code  is  shown  in  Figure  3.2  on  page  64).  Calling 
within-task  to  modify  the  operation  of  a  teisk  which  is  already  running 
marks  the  task  data  structure  to  indicate  the  work  that  must  be  performed, 
and  it  is  the  responsibility  of  the  ta.sk  when  it  next  saves  its  state  away  to 
arrange  to  perform  that  work  when  the  task  is  next  activated. 

The  routine  store-my-state,  shown  in  Figure  4.3  is  provided  to  sup¬ 
port  this  operation.  It  allows  scheduler  routines  to  specify  the  state  to  be 
used  when  the  task  noimally  regains  control  (the  argument  state),  and  ad¬ 
ditional  work  to  be  performed  on  the  task  data  structure  while  it  is  locked 
(while-locked).  As  you  can  see,  it  locks  the  current  task  data  structure 
and  then  stores  either  the  specified  state  or  a  procedure  that  first  executes 
the  work  specified  by  a  call  to  within-task  as  the  work  to  be  performed 
when  the  task  is  next  activated. 

One  other  common  packaging  of  this  first  operation  is  provided  by 
release-task,  shown  in  Figure  4.4.  In  this  case,  the  intention  is  to  re- 
lea.se  the  current  task  for  (possibly  parallel)  execution  and  then  execute 
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(define  (stora-my-state  state  while-locked) 

(let  ((my-task  (current-task))) 

(with-task-locked  my-task 

(lambda  (am-I-runnable?)  ;  (1) 

(if  am-I-runnable? 

(begin 

(set-task. code!  my-task 

(if  (eq?  (task. status  my-task)  ’WITHIN-TASK) 

(let  ((within-task-code  (task. code  my-task))) 
(lambda  (wake-up)  ;  (2) 

(within-task-code  wake-up) 

(state  wake-up))) 
state))  ;  (3) 

(while-locked  my-task)))))))  ;  (4) 


Notes: 

1.  Find  and  lock  the  current  task  data  structure. 

2.  If  the  task  is  expected  to  continue  running  but  has  been  marked  for  special  handling 
by  within-task  (see  Figure  3.2  on  page  64),  then  when  the  task  next  awakens  it 
must  first  execute  the  code  specified  in  the  call  to  within-task  and  then  continue 
on  to  its  ordinary  computation. 

3.  Under  ordinary  circumstances,  the  state  to  be  stored  is  just  the  state  specified  by 
the  caller. 

4.  If  the  task  will  continue  to  run,  call  the  user-specified  procedure  while  the  task  data 
structure  is  still  locked. 


Figure  4.3:  Saving  state  for  future  execution:  Store-My-State 
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(define  (release-task  thunk) 
(call-with-current-continuation 
(lambda  (my-state) 

(store-my-state  my-state 
(lambda  (my -task) 

(set-current-task!  ’STATE-SAVED) 
(set-task . status !  my-task  ’RUNNABLE) 
(put-work  my-task))) 

(thunk) ) ) ) 


Figure  4.4:  Relinquishing  the  processor:  Release-Task 


some  other  code  while  the  processor  is  temporarily  not  performing  an  ex¬ 
isting  task.  The  only  non-obvious  part  of  Release-task  is  why  it  must 
be  provided  with  the  code  to  be  executed.  Notice,  however,  that  the  code 
is  executed  as  part  of  the  procedure  called  by  the  call-with-current- 
continuation  operation.  Thus  it  is  executed  by  the  calling  task,  but  noi 
when  that  task  is  resumed  by  calling  the  my-state  continuation. 

The  second  operation,  choosing  a  task  to  perform,  is  most  often  deferred 
to  the  underlying  machine,  using  the  primitive  get-work  to  select  the  task: 


(define  (next) 

(Set-Current-Task I  ’ WAITING-FOR-WORK) 
(run  (get-work))) 


The  third  operation,  activating  a  chosen  task,  is  the  most  complicated. 
This  job  is  handled  by  the  procedure  run,  shown  in  Figure  4.5.  It  consists 
mostly  of  routine  housekeeping  activities.  The  task  being  activated  is  first 
locked  and  tested  to  see  if  it  is  actually  runnable.  If  so,  the  status  is  changed 
to  running  and  the  code  and  wake-up  value  are  extracted  from  the  task  data 
structure.  The  task  is  then  unlocked,  and  cither  the  code  is  activated  with 
the  appropriate  wake-up  value  cis  its  argument  or  (if  the  task  turned  out 
not  to  be  runnable)  an  alternative  task  is  chosen  using  next. 
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(define  (run  task) 

(define  what-to-actually-do 
(With-Task-Locked  task 
(lambda  (Still-Runnable?) 

(if  Still-Runnable?  ;  (1) 

(let  ((code-for-new-task  (task. code  task)) 

(wake-up-value  (task. wake-up-value  task))) 
(set-task. status!  task  'RUNNING) 
(set-task.wake-up-value!  task  ’()) 

(Set-Current-Task!  task) 

(lambda  ()  ;  (2) 

(code-for-new-task  wakeup-value) ) ) 
next))))  ;  (3) 

(what-to-actually-do))  ;  (4) 

Notes: 

1.  Test  the  task  to  see  if  it  is  actually  runnable. 

2.  If  the  task  is  runnable,  this  procedure  will  restart  it. 

3.  If  the  task  is  not  runnable,  this  procedure  will  select  an  alternate  task  and  start  it 
instead. 

4.  Actually  call  the  procedure  chosen  in  steps  2  and  3  above. 

Figure  4.5;  Activating  a  chosen  task:  the  run  procedure 
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4.2.3  Other  Utility  Routines 

There  are  a  handful  of  other  utility  routines  that  are  referenced  later  in 
this  chapter. 

•  A  task  data  structure  and  its  related  placeholder  can  be  created  using 
(Make-Task),  which  makes  the  pair  simultaneously  and  supplies  a 
standard  set  of  default  values  for  all  of  the  information  req'”red. 


(define  (immutable?  placeholder) 

(eq?  (placeholder .known?  placeholder)  #t)) 

(define  (undetermined?  placeholder) 

(eq?  (placeholder .known?  placeholder)  #f)) 

(define  (determined?  placeholder) 

(not  (undetermined?  placeholder))) 

(define  (mutable?  placeholder) 

(and  (determined?  placeholder) 

(not  (immutable?  placeholder)))) 


Figure  4.6:  Tri-state  Flag  Representation 


#  The  three  possible  states  of  the  determined?  slot  of  a  placeholder  are: 
#t  indicating  that  the  placeholder  has  an  immutable  value;  #f  indi¬ 
cating  that  it  has  no  value  at  all;  and  anything  else  indicates  that  the 
value  is  mutable.  This  is  captured  in  the  three  procedures  shown  in 
Figure  4.G.  Notice  that  the  placeholder  must  be  locked  in  order  to 
safely  perform  the.se  operations. 

•  A  task  that  has  l)een  waiting  is  activated  using  the  procedure  acti¬ 
vate  shown  in  Figure  4.7.  This  procedure  tests  whether  the  task  is 
still  runnable  and  is  in  fact  waiting  for  the  condition  that  has  occurred 
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(define  (activate  task  test  wake-up-value) 

(With-Task-Locked  task  (lambda  (task-rimnable?) 

(if  (and  task-runnable?  (test  (task. status  task))) 
(begin 

(set-task. uaiting-for!  task  ’()) 

( set- t ask . status !  task  ’RUNNABLE) 

(set-task. wake-up-value!  task  wake-up-value) 
(put-work  task)))))) 


Figure  4.7:  Activating  a  waiting  task 


1^ 


(as  indicated  by  the  test).  It  then  updates  the  task  data  structure 
and  releases  it  for  distribution  using  the  underlying  machine  opera¬ 
tion  put-work. 

•  The  procedure  Saving-State,  shown  in  Figure  4.8  was  mentioned 
earlier  in  Section  3.1.3.  It  allows  a  task  to  save  its  state  away  and 
then  execute  a  selected  piece  of  code  (thunk).  The  code  is  run  in  a 
continuation  that  is  part  of  the  root  of  garbage  collection  and  not  as 
part  of  the  task  that  called  saving-state.  This  permits  the  garbage 
collector  to  reclaim  the  originating  task  if  necessary.  When  the  code 
finishes  execution,  another  task  is  selected  for  execution  rather  than 
returning  to  the  original  task.  A  good  way  of  thinking  about  saving- 
state  is  that  it  performs  a  task  switch  (into  a  non-existent  task)  and 
executes  the  thunk  in  the  new  task. 

•  Two  “ugly”  procedures,  Current-Task  and  Set-Current-Task  I ,  are 
provided  to  keep  track  of  the  task  that  is  currently  executing.  These 
maintain  data  that  is  private  to  the  processor  and  are  used  to  im¬ 
plement  the  higher  level  concepts  of  task-private  data  as  described  in 
Section  3,3.5.  The  author  is  too  embarrassed  to  reveal  the  implemen¬ 
tation. 

•  Weak-list->list  converts  a  list  composed  of  weak  cons  cells  into 
one  composed  of  ordinary  cons  cells. 


-  *  -  - 
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(define  (saving-state  thunk) 

(release-task  (lambda  ()  ;  (1) 

(within-control-point  ;  (2) 

the - er r or- cont inuat ion 
(lambda  ()  (thunk)  (next)))))) 


Notes: 

1.  Relezise  the  current  task  for  potential  parallel  execution, 

2.  Begin  execution  within  the-error-cont  inuat  ion  which  was  created  at  system  boot 
time,  and  does  not  reference  any  other  continuations. 

Figure  4.S:  Code  for  Saving-State 


•  A  package  of  queue  manipulation  routines,  including  enqueue,  de¬ 
queue,  and  queue-contents.  Using  these  the  operation  add-to- 
waiting-queue !  is  implemented: 

(define  (add-to-waiting-queue!  placeholder  task) 

(enqueue  (placeholder .waiting-queue  placeholder)  task)) 


4.3  Task  Creation  and  Termination 

Creating  a  task  in  MultiScheme  really  has  four  steps:  create  a  continua¬ 
tion,  create  a  task  (i.e.  the  task  data  structure),  create  a  placeholder,  and 
schedule  the  running  and  newly  created  tasks  for  (possibly  parallel)  execu¬ 
tion.  Each  of  these  can  be  performed  independently  and  then  combined  to 
provide  specialized  handling  of  unusual  cases.  Tasks  are  normally  created, 
however,  by  using  the  future  macro. 

This  macro  has  one  required  argument,  the  expression  to  be  executed  in 
parallel,  and  an  optional  policy  used  to  schedule  the  parent  and  child  tasks. 
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Thus,  the  macro  expands 

(+  (future  expression  policy)  (future  expression2) ) 
into  • 

(+  (spawn-task  (lambda  ()  expression)  policy) 

(spawn-task  (lambda  ()  expression2)  p2u:ent-gets-priority) 

The  remainder  of  this  section  consists  of  a  description  of  the  Spawn- 
Task  procedure  (Section  4.3.1),  alternatives  to  this  standard  method  of 
task  creation  (Section  4.3.2),  and  finally  the  handling  of  task  termination 
(Section  4.3.3). 

4.3.1  Ordinary  Task  Creation 

As  described  above,  most  of  the  work  of  creating  a  task  is  ordinarily  carried 
out  by  Spawn-Task.  This  is  merely  a  standard  way  of  using  the  four  steps 
mentioned  above. 

Create  a  continuation. 

One  of  the  parts  of  the  task  data  structure,  described  above,  is  the 
code  it  is  to  execute.  Spawn-Task  creates  a  continuation  for  this  pur¬ 
pose  (see  the  discussion  below).  The  continuation  could  be  expressed 
as  the  procedure: 

(lambda  (ignored-argument) 

(termination-continuation  (thunk) ) ) 

In  this  procedure,  thunk  is  the  first  argument  to  Spawn-Task,  the  pro¬ 
cedure  created  from  the  expression  by  expanding  the  future  macro. 
The  termination-continuation  is  a  specific  (primitive)  continua¬ 
tion  that  indicates  the  end  of  a  task.  The  handling  of  this  continua¬ 
tion  is  described  in  Section  4.3.3. 

Create  a  task  data  structure. 

The  space  for  the  data  structure  that  was  described  in  Section  4.1.2 
must  be  allocated  and  initialized.  The  code  slot  is  filled  with  the 
continuation  created  in  the  previous  step.  The  goal  slot  is  filled  with 
the  placeholder  created  in  the  following  step. 
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Create  a  placeholder. 

This  acts  as  the  goal  of  the  task  being  created.  It  will  receive  the 
value  computed  by  that  task  when  the  termination  continuation  is 
reached.  The  task  created  in  the  previous  step  is  made  the  motivated 
task  of  the  placeholder. 

Schedule  the  tasks. 

Spawn-Task  calls  the  user-supplied  policy  routine  to  schedule  the  cur¬ 
rent  (spawning)  task  and  the  newly  created  task.  The  default  routine, 
Parent-Gets-Priority  (shown  in  Section  4.3.2),  releases  the  new 
task  for  potentially  parallel  execution.  The  value  returned  by  Spawn- 
Task  to  the  task  that  called  it  is  the  newly  created  placeholder. 

The  decision  to  use  a  continuation  (rather  than  a  procedure)  for  the  initial 
code  of  a  task  is  not  completely  arbitrary.  If  a  procedure  is  used  the  task 
switch  code  in  run  (see  Figure  4.5)  that  activates  the  newly  created  task 
would  be  nothing  more  than  a  procedure  call.  But  procedure  call  includes 
passing  an  implicit  continuation  for  use  when  a  value  is  returned,  and  this 
continuation  will  in  some  way  reference  the  task  that  made  the  procedure 
call.  This  prevents  the  garbage  collector  from  reclaiming  that  task  as  long 
as  the  newly  created  task  is  in  existence. 

In  implementation  terms,  which  may  be  easier  to  understand,  proce¬ 
dure  call  is  handled  by  using  a  single  stack  to  hold  continuations  (return 
addresses).  If  the  initial  code  for  a  task  were  simply  a  procedure  then  the 
stack  used  for  the  new  task  when  it  first  runs  would  be  the  same  as  the  stack 
of  the  task  that  was  relinquishing  the  processor.  This  works  perfectly  well 
but  leads  to  a  form  of  cactus  stack  implementation  that  has  the  garbage 
collection  problem  mentioned  above. 

By  explicitly  building  a  continuation,  however,  task  switch  becomes  the 
same  as  invoking  a  continuation  that  does  not  implicitly  reference  the  old 
ta.sk.  The  contiimatioii  created  when  the  task  is  created  is  an  initial  stack 
frame  and  task  switch  (now  throw)  causes  the  stack  to  be  switched  as  well. 

4.3.2  Alternative  Ways  to  Create  a  Task 

The  task  creation  code  is  modularized  into  the  four  steps  described  above. 
Art\ially  utilizing  these  individual  components  is  unusual  since  the  flexibil¬ 
ity  available  u.sing  the  second  (policy)  argument  to  Spawn-Task  is  sufficient 
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for  most  problems.  To  make  this  power  easily  available,  two  alternative 
policies  ^lre  included  in  the  scheduler  package  along  with  the  default  policy. 
The  standard  policy,  Parent-Gets-Priority  is  very  efficient: 

(define  (Parent-Gets-Priority  new-task) 

(put-work  new-task) 

>  CHILD-QUEUED-FOR-EXECUTION) 

This  policy  gives  processing  priority  to  the  parent  task.  That  is,  the  task 
that  calls  Spawn-Task  continues  to  run  after  the  call,  while  the  task  which  is 
created  is  scheduled  for  (possibly  parallel)  execution.  Since  the  task  and  its 
associated  placeholder  have  been  made  and  initialized  by  Spawn-Task,  all 
that  must  be  done  is  to  make  the  new  task  available  for  computation.  This 
is  done  using  the  underlying  task  distribution  mechanism,  implemented  by 
put-work.  (In  this,  as  in  the  other  policies,  the  value  returned  by  the  policy 
is  ignored  by  Spawn-Task  but  is  useful  in  debugging  the  scheduler  itself.) 

Halstead  argues,  in  his  overview  of  Multilisp[26],  that  this  standard 
policy  can  lead  to  undesired  performance  characteristics  as  a  system  reaches 
saturation.  He  suggests  a  strategy  in  which  the  parent  task  is  deferred  while 
the  child  task  immediately  begins  execution.  This  is  implemented  using  the 
Child-Gets-Priority  policy. 

(define  (Child-gets-priority  new-task) 

(release-task  (lambda  ()  (run  new-task)))) 

The  third  policy,  Delay-Policy,  marks  the  spawned  task  as  delayed  and 
does  not  release  it  for  parallel  execution. 

(define  (Delay-policy  new-task) 

(set-task . status !  new-task  ’DELAYED) 

’OK-I-DELAYED-IT) 


Instead,  the  first  task  that  touches  the  goal  placeholder  associated  with 
the  newly  created  task  will  release  that  task  for  execution  (see  Section  4.4 
below). 

In  addition  to  these  three  policies,  there  is  one  other  case  that  occurs 
sufficiently  often  to  be  provided  standardized  support.  This  is  the  ability  to 
wait  for  the  first  of  a  number  of  placeholders  to  return  a  value.  This  ability, 
implemented  by  the  procedures  disjoin  and  await-f  irst  of  the  scheduler, 
is  the  key  to  the  aunb  and  fair-merge  procedures  discussed  in  section  2.5. 
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(define  (disjoin  .  Placeholders)  ;  (1) 

(let  ((My-Task  (Hake-Task))) 

(let  ( (My-Placeholder  (task. goal  My-Task))) 
(set-task . status !  My-Task  'DISJOIN) 

(set-task. waiting-for!  My-Task  Placeholders) 
(set-task . code !  My-Task 

(lambda  (awakened-value)  ;  (2) 

(determine!  My-Placeholder  awakened-value) 
(next) ) ) 

(for-each  ;  (3) 

(lambda  (Placeholder) 

(add-to-waiting-queue !  Placeholder  My-Task)) 
Placeholders) 

My-Placeholder)))  ;  (4) 


Notes: 

1  .4s  explained  in  the  text,  this  code  does  not  deal  with  a  number  of  important 

possibilities. 

2.  Code  to  be  run  when  this  newly  created  task  is  activated  (i.e.  when  one  of  the 
placeholder  receives  a  value).  This  is  one  of  two  major  race  conditions  that  the 
complete  ver.sion  handles.  If  more  than  one  task  completes,  My-Placeholder  may 
already  have  a  value  when  this  code  is  run. 

3.  Enqueue  this  task  on  the  waiting-queue  of  each  of  the  placeholders.  This  is  the 
second  of  the  major  race  conditions.  In  the  process  of  enqueuing,  it  may  be  discov¬ 
ered  that  one  of  the  placeholders  already  has  a  value  which  must  then  be  returned 
instantly. 

4.  The  value  returned  by  disjoin  is  the  placeholder  that  will  ultimately  receive  the 
value  of  the  first  computed  placeholder. 


Figure  4.9:  Simplified  Code  for  disjoin 
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The  actual  code  for  these  procedures  is  complicated  because  it  must  deal 
with  the  possibility  that  one  of  the  placeholders  has  already  received  a 
value  before  the  operation  has  been  completed,  and  because  more  than 
one  of  the  placeholders  may  eventually  receive  a  value.  Figure  4.9  shows  a 
much  simpler  version  that  does  not  deal  with  these  problems  (the  footnotes 
to  the  figure  explain  the  most  important  omissions).  Recall  that  disjoin 
itself  returns  a  placeholder  rather  than  actually  waiting  for  the  value  to  be 
known. 

This  simplified  version  works  by  creating  a  task  and  its  corresponding 
placeholder  (goal)  using  Make-Task.  The  purpose  of  this  new  task  is  to 
propagate  the  value  of  the  appropriate  placeholder  (the  first  one  that  re¬ 
ceives  a  value)  out  to  its  own  goal.  The  code  to  be  performed  when  this 
new  task  is  awakened  is  supplied  as  an  explicit  procedure**.  This  is  very 
similar  to  the  processing  of  the  normal  case,  except  that  the  task  heis  a  list 
of  placeholders  (rather  than  a  single  placeholder)  for  which  it  is  waiting 
and  it  is  enqueued  on  each  of  these  placeholders. 

When  any  of  the  placeholders  for  which  this  task  is  waiting  receives 
a  value  (see  Section  4.5),  that  placeholder  is  stored  in  this  task’s  wake-up 
value  slot  and  the  task  is  made  available  for  execution.  When  the  task  is 
activated  (using  the  run  procedure  described  in  Section  4.2.2)  the  procedure 
stored  in  the  code  slot  will  be  passed  this  wake-up  value.  The  procedure 
will  propagate  it  to  the  placeholder  created  by  the  call  to  disjoin,  and 
then  call  next  (see  Section  4.2.2)  to  release  the  processor  and  find  another 
task. 

The  use  of  a  task  to  propagate  the  value  of  the  appropriate  disjunct 
may  seem  unusual,  but  it  improves  the  modularity  of  the  scheduler  code 
itself.  This  work  could  have  been  made  part  of  the  determine!  code,  but 
this  organization  allows  determine!  to  simply  awaken  tasks  in  a  standard 
manner.  Determine!  is  never  required  to  do  ^my  specialized  processing  on 
behalf  of  the  tasks  it  awakens. 


‘'The  choice  of  a  procedure  rather  than  a  continuation  here  is  somewhat  arbitrary.  It  is 
ea.sier  to  write  as  shown.  Furthermore,  the  procedure  will  relinquish  the  processor  and  be 
itself  garbage  collected  almost  instantly  so  that  the  garbage  collection  problem  mentioned 
earlier  is  not  an  issue. 
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4.3.3  Task  Termination 

As  mentioned  in  Section  4.3.1,  there  is  a  primitive  continuation  that  denotes 
the  termination  of  a  task.  As  with  any  continuation,  it  receives  a  value  that 
is  treated  as  the  value  for  the  goal  of  that  task.  To  make  modifications  to 
the  system  simpler,  the  handling  of  this  continuation  (unlike  most  primitive 
continuations)  is  reflected  back  into  the  scheduler  as  a  call  to  a  procedure. 
The  standard  procedure  is  quite  simple  since  it  runs  as  part  of  the  task 
that  is  terminating: 

(define  (end-of-computation-handler  value) 

(determine!  (task. goal  (currant-task))  value) 

(next) ) 

This  merely  stores  the  final  value  into  the  goal  and  then  locates  and 
activates  the  next  available  task.  Notice  that  by  simply  calling  next  without 
saving  its  own  state,  this  task  relinquishes  the  processor  and  will  not  be 
reactivated^. 

4.4  Suspending  a  Task 

There  are  three  ways  in  which  a  teisk  can  relinquish  the  processor.  It  can 
explicitly  relinquish  the  processor  using  the  reschedule  procedure  (men¬ 
tioned  earlier  in  Section  3.1.4).  An  interrupt  can  occur  and  cause  the  pro¬ 
cessor  to  be  relinquished  (for  example,  the  initiation  of  a  garbage  collection 
or  a  clock  interrupt).  Finally,  and  most  commonly,  the  task  can  attempt 
to  touch  a  placeholder  that  does  not  yet  have  a  value. 

With  the  utility  procedures  described  earlier  it  should  be  easy  to  see 
how  the  first  operation  is  performed: 

(define  (reschedule) 

(release-task  next)) 

Garbage  collection  initiation  was  described  in  Section  3.1.1.  The  actual 
code  to  implement  it  is  very  similar  to  that  of  pause-everything  discussed 

is  possible,  of  course,  to  write  code  that  saves  the  state  of  the  task  prior  to  ending 
the  computation.  The  results  of  such  an  action  can  be  predicted  eaisily  enough,  and  miglit 
even  be  useful  in  some  cases.  But  this  is  so  far  out  of  the  ordinary  as  to  be  almost  certainly 


an  error. 


4.4.  SUSPENDING  A  TASK 


109 


in  Section  3.1.3.  Timer  interrupts  are  handled  by  calling  reschedule  as 
part  of  the  interrupt  handler. 

The  remainder  of  this  section  is  devoted  to  the  third  problem,  touching 
a  placeholder.  When  placeholders  were  introduced  in  Section  2.1  it  was 
stated  that  they  “can  be  used  to  denote  an  object  whose  value  is  not  yet 
known,”  and  that  the  scheduler  is  responsible  for  handling  an  attempt  to 
touch  a  placeholder  which  does  not  yet  have  a  value.  This  mechanism  has 
two  parts,  one  implemented  in  the  machine  underlying  the  MultiScheme 
system,  and  the  other  as  part  of  the  scheduler. 

The  underlying  machine  is  responsible  for  both  detecting  and  handling 
the  simple  cases  related  to  placeholder  objects.  There  are  three  different 
ways  in  which  a  placeholder  can  be  initially  noticed: 

1.  The  “microcode”  for  certain  operations  explicitly  touches  objects 
that  they  manipulate.  If  the  object  is  not  a  placeholder,  or  the  place¬ 
holder  has  a  value,  the  microcode  will  retrieve  the  correct  value  and 
the  operation  proceeds  unimpeded.  The  operations  are  all  carefully 
written,  however,  so  that  if  a  placeholder  is  encountered  that  does  not 
have  a  value  the  operation  can  be  stopped  and  restarted  at  a  later 
time.  The  operation  gracefully  backs  out  and  returns  the  state  of 
the  system  to  what  it  was  before  the  operation  began.  It  then  per¬ 
forms  a  call  to  the  scheduler’s  await-placeholder  operation.  The 
result  is  as  though  the  user  had  written  a  call  to  touch  (which  is  im¬ 
plemented  using  await-placeholder)  of  the  appropriate  placeholder 
immediately  prior  to  the  call  to  the  operation. 

2.  Many  primitive  operations  normally  type-check  their  arguments  for 
validity  before  doing  any  processing.  For  those  operations  that  do 
not  permit  an  operand  to  be  a  placeholder  (for  example,  arithmetic 
operations  restricted  to  numeric  data  types),  the  normal  error  han¬ 
dling  mechanism  of  MIT  Scheme  would  cause  the  primitive  to  grace¬ 
fully  back  out  (just  as  in  the  previous  case)  and  then  invoke  an  error 
handling  procedure.  In  MultiScheme,  the  microcode  for  these  error 
handlers  tests  for  placeholders  and  restarts  the  primitive  automati¬ 
cally  (i.e.  without  any  form  of  trap  into  Scheme  code)  if  the  erroneous 
argument  is  a  placeholder  that  has  a  value.  If  the  argument  is  a  place¬ 
holder  that  does  not  yet  have  a  value  then  instead  of  invoking  one 
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of  the  Sdieme  error  handling  procedures  it  invokes  the  scheduler’s 
await -placeholder  procedure. 

3.  Compiled  code  contains  calls  to  the  primitive  operation  touch  when¬ 
ever  it  must  ensure  that  cui  object  (argument  to  an  in-line  coded 
primitive,  predicate  of  a  conditional,  or  function  to  be  applied)  is  not 
a  placeholdei  and  cannot,  at  compile  time,  deduce  this.  Touch,  which 
is  one  of  the  operations  described  in  case  1,  merely  tests  its  operand 
to  see  if  it  is  a  placeholder.  If  it  is  not,  the  operand  is  returned.  If  it 
is  a  placeholder  with  a  value,  that  value  is  returned.  The  net  result 
is  that  an  attempt  to  use  a  placeholder  whose  value  is  already  known 
will  proceed  unimpeded,  but  one  whose  value  is  still  undetermined 
will  cause  a  call  to  the  await-placeholder  procedure.  (The  exact 
placement  of  these  calls  to  touch  is  a  topic  for  further  investigation. 
Placing  them  earlier  in  the  code  can  frequently  make  the  code  more 
efficient  but  it  reduces  the  potential  for  pEirallelism.) 

In  each  case,  the  underlying  machine  handles  placeholders  that  have  al¬ 
ready  received  a  value  but  calls  the  await-placeholder  procedure  to  han¬ 
dle  placeholders  that  do  not  have  a  value.  The  job  of  await-placeholder, 
then,  is  to  save  the  state  of  the  current  task  (if  it  needs  to  continue  run¬ 
ning)  and  enqueue  this  task  on  the  waiting-queue  of  the  placeholder.  It  then 
releases  the  processor  by  calling  next. 

The  code  for  await-placeholder  is  shown  in  Figure  4.10.  The  following 
description  of  its  operation  is  keyed  to  the  numbers  in  the  figure. 

1.  Create  a  continuation,  me,  that  holds  the  state  of  the  current  task. 
Attempt  to  lock  the  placeholder  for  which  the  taisk  is  waiting. 

2.  If  the  placeholder  couldn’t  be  locked,  just  resume  the  current  task. 
This  can  occur  if  the  placeholder  has  received  an  immutable  value 
prior  to  reaching  this  point  in  the  code.  The  placeholder  would  be 
subject  to  the  splicing  operation  (mentioned  in  Section  4.1.1)  during 
variable  reference  or  garbage  collection.  After  the  lock  is  acquired 
this  splicing  will  no  longer  occur. 

3.  If  the  placeholder  has  a  value,  unlock  the  placeholder  and  resume 
the  current  task.  This  can  occur  if  the  placeholder  has  received  a 
mutable  value  before  reaching  this  point  in  the  code.  In  this  case,  the 
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(define  (await-placeholder  placeholder) 
(call-with-current-continuation 
(lambda  (me)  ;  (1) 

(With-Placeholder-Locked  placeholder 
(lambda  (waiting-for-a-placeholder?) 

(cond  ((not  waiting-for-a-placeholder?) 

(me  ’RESUME-COMPUTATION))  ;  (2) 

((determined?  placeholder) 
(unlock-placeholder!  placeholder) 

(me  ’RESUME-COMPUTATION)))  ;  (3) 

(activate  ;  (4) 

(placeholder .motivated-task  placeholder) 

(lambda  (status) 

(or  (eq?  status  ’DELAYED)  (eq?  status  ’PAUSED))) 
’ NO-RELEVANT-WAKE-UP-VALUE) 

(store-my-state  me  (lambda  (My-Task)  ;  (5) 

(set-task . status !  My-Task  ’WAITING) 

(set-task. waiting-f or!  My-Task  placeholder) 
(add-to-waiting-queue!  placeholder  My-Task))))) 
(next))))  ;  (6) 


Soo  text  beginning  on  page  110  for  footnotes. 


Figure  4.10:  Code  for  Await-Placeholder 
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plact'hoUler  is  not  subject  to  splicing  so  it  will  have  been  locked,  but 
there  is  no  nt'ed  to  await  the  arrival  of  a  value. 

4.  Activate  the  task  that  is  calculating  the  value  for  the  placeholder  if 
it  is  inactive.  This  arises  either  because  the  placeholder  was  created 
by  the  delay-policy  (hence  the  task  has  status  delayed),  or  because 
the  task  has  been  suspended  by  pause-everything  (see  Section  4.6), 
and  the  task  has  status  paused. 

5.  At  this  point,  the  task  will  definitely  be  releasing  control  of  the  pro¬ 
cessor.  The  state  of  the  task,  me  from  step  1,  is  saved  away  in  the 
current  task  data  structure.  Mark  the  task  as  waiting  and  add  it  to 
the  waiting-queue  of  this  placeholder. 

6.  Unlock  the  placeholder  (by  exiting  the  with-placeholder-locked 
procedure).  Find  another  task  and  start  executing  it.  Notice  that  this 
call  to  next  is  not  executed  when  control  returns  to  the  original  task 
using  the  me  continuation  created  in  step  1  (count  the  parentheses. . .). 

4.5  Storing  the  Value  of  a  Placeholder 

Storing  a  value  into  a  placeholder  is  a  straightforward  operation,  al¬ 
though  the  details  are  somewhat  complicated.  The  essential  work  is  to 
store  the  value  into  the  placeholder  data  structure  and  activate  any  tasks 
that  may  have  been  waiting  for  this  value  to  appear.  The  detailed  code  is 
shown  in  Figure  4.11.  The  following  notes  describe  the  fine  details  of  its 
operation.  They  are  geared  to  the  numbers  appearing  in  the  figure.  Most 
of  the  complexity  comes  from  the  need  to  keep  the  placeholder  locked  for 
as  short  a  time  period  as  possible,  and  the  possibility  of  a  race  if  two  tasks 
attempt  to  supply  values  to  the  placeholder  nearly  simultaneously. 

1.  The  auxiliary  procedure  update-placeholder!  actually  makes  the 
changes  necessary  to  the  placeholder  data  structure  to  reflect  the  fact 
that  it  now  has  a  value. 

2.  What-to-do  will  contain  one  of  three  procedures  to  be  performed  after 
th('  placeholder  is  unlocked,  in  step  7.  The  three  procedures  are  (a) 
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(define  (determine!  placeholder  value  allou-mutations?) 

(define  (update-placeholder!)  ;  (1) 

(set-task . status !  (placeholder .motivated-task  placeholder) 
’DETERMINED) 

(set-placeholder .value!  placeholder  value) 

(set-placeholder . determined? !  placeholder 
(if  allow-mutations?  'MUTABLE  it))) 

(define  uhat-to-do  ;  (2) 

(With-Placeholder-Locked  placeholder 
(lambda  (still-a-placeholder?) 

(cond  ((or  (not  still-a-placeholder?) 

(immutable?  placeholder)) 

(lambda  ()  ;  (3) 

(error  "Immutable  Placeholder"  placeholder))) 
((undetermined?  placeholder) 

(let  ((waiters 

(queue-contents 

(placeholder .waiting-queue  placeholder) ) ) ) 
(update-placeholder!)  ;  (4) 

(lambda  ()  ;  (5) 

(for-each 
(lambda  (task) 

(activate  task 

(lambda  (status) 

(or  (eq?  status  ’WAITING) 

(eq?  status  ’DISJOIN))) 
placeholder) ) 

waiters) ) ) ) 

(else  (update-placeholder!)  ;  (6) 

(lambda  ()  ’OK)))))) 

(what-to-do)  ;  (7) 

value)  ;  (8) 

S<'c  toxt  beginning  on  page  112  for  footnotes. 

Figure  4.11:  Code  for  determine ! 
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an  error  procedure,  step  3;  (b)  awaken  the  waiting  tasks,  step  5;  and 
(c)  do  nothing,  step  6. 

3.  With  the  placeholder  locked,  test  whether  it  already  h8ts  an  immutable 
value.  If  so,  return  a  procedure  that  will  cause  an  error  in  step  7. 

4.  If  the  placeholder  did  not  previously  have  a  value,  remember  the  tasks 
that  are  waiting  for  the  value  of  this  placeholder  (in  waiters)  and 
then  update  the  placeholder. 

5.  Since  the  placeholder  didn’t  have  a  value  before,  in  step  7  we  must 
activate  (see  Figure  4.7)  each  task  that  is  waiting  for  this  place¬ 
holder.  The  activation  test  permits  only  tasks  that  are  waiting  for  a 
placeholder  (status  waiting  or  disjoin)  to  be  awakened. 

6.  If  the  placeholder  previously  had  a  mutable  value,  then  it  can’t  have 
a  queue  of  tasks  waiting  for  its  value  to  appear.  Thus,  in  step  7  we 
don’t  need  to  take  any  special  action. 

7.  Now  that  the  placeholder  is  unlocked,  perform  whatever  work  is  nec¬ 
essary. 

S.  The  value  returned  by  determine!  is  (arbitrarily)  the  value  that  has 
been  given  to  the  placeholder. 

4.6  Single  Task  Interludes 

The  final,  and  most  complicated,  operation  supported  by  the  scheduler  is 
the  ability  to  switch  from  a  parallel  processing  mode  where  many  tasks 
are  simultaneously  active  to  one  in  which  only  a  single  task  is  active.  This 
operation,  embodied  in  the  procedure  pause-everything,  has  already  been 
discussed  in  Section  3.1.3,  and  the  code  is  shown  in  Figure  3.1  on  page  62. 
This  secticm  presents  the  underlying  support  routine  that  was  omitted  in 
that  earlier  version®. 

.\s  mentioned  in  the  earlier  discussion  of  pause-everything  its  job  is 
to  suspend  all  other  tasks  on  the  system.  It  returns  as  its  value  an  object 

'’A  procedure  with  structure  very  similar  to  pause-everything  but  without  the  elab¬ 
orate  returned  object  is  used  to  initiate  garbage  collection. 
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(define  (make-returned-object  the-queue) 

(lambda  (message) 

(cond  ((eq?  message  ‘ANY-TASKS?)  ;  (1) 

(and  (not  (eq?  the-queue  #t)) 

(not  (eq?  the-queue  ’())))) 

((eq?  message  ’RESTART-TASKS)  ;  (2) 

(if  (eq?  the-queue  it) 

(error  "Attempt  to  re-use  a  pause  object!") 

(begin 

(for-each  ;  (3) 

(lambda  (task) 

(activate  task 

(lambda  (status)  (eq?  status  ’PAUSED)) 
’NO-RELEVANT-WAKE-UP-VALUE) 
the-queue) ) 

(set!  the-queue  it)))) 

((eq?  message  ’THE-TASKS)  ;  (4) 

(if  (eq?  the-queue  it)  ’()  the-queue)) 

(else  (error  "Pause:  unknown  message"  message))))) 

Notes: 

1 .  Code  to  handle  the  Any-Tasks?  message. 

2.  Code  to  handle  the  Restart-Tasks  message. 

3.  If  this  is  the  first  time  the  restart-tasks  message  is  received,  any  tasks  that  are 
still  paused  are  activated.  Notice  that  the  code  for  await-placeholder  shown  in 
F'igure  4.10  will  have  activated  any  of  these  tasks  that  were  touched  after  the  call 
to  pause-everything.  Hence,  the  test  here. 

4.  Code  to  handle  the  The-Tasks  message. 

Figtire  4.12:  Make-Returned-Object,  support  for  pause-everything 
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tliat  encapsulates  these  other  tasks  through  a  “message  passing”  interface. 
The  returned  object  (implemented  in  MIT  Scheme  as  a  procedure  of  one 
argument,  the  message)  accepts  three  messages. 

Auy-Tasks? 

Returns  a  boolean  answer  of  #t  if  there  were  other  tasks  running  at 
the  time  of  the  call  to  pause-everything  and  they  have  not  yet  b(‘(>n 
restarted. 

The-Tasks 

Returns  a  list  of  the  tasks  that  were  suspended  by  the  call  to  pause- 
everything  provided  they  have  not  yet  been  restarted. 

Restart-Tasks 

.Activates  the  tasks  that  were  suspended  by  the  call  to  pause-every¬ 
thing  by  releasing  them  to  the  underlying  task  distribution  mecha¬ 
nism.  It  can  be  called  only  once. 

The  procedure  Make-Returned-Obj ect,  shown  in  Figure  4.12.  creates  the 
message  accepting  object  that  will  be  returned  by  pause-everything. 

4.7  Summary 

The  scheduler  is  the  “heart”  of  the  MultiScheme  svstem.  and  is  designed  as 
;i  fie.xible  and  ext<'nsible  nu'chanisin  for  supporting  a  variety  of  experiments. 
The  primary  data  structures  of  the  scheduler,  the  i)laceholder  and  the  task 
are  descriljed  in  Section  4.1.  Using  these  data  structures,  a  variety  of  ways 
of  creating,  suspending,  and  managing  tasks  are  described. 

Most  (if  the  material  presented  is  very  much  “nuts  and  Iiolts  engineer¬ 
ing"  but  serves  to  demonstrate  the  ease  with  which  the  ])arallel  processing 
-upport  e;in  be  descrit>ed  within  the  existing  language  framework. 
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The  earlier  chapters  have  described  the  structure  of  the  MultiScheme  sys¬ 
tem  and  a  number  of  the  important  design  choices.  This  chapter  consists 
of  three  programs  that  exploit  parallelism  and  demonstrate  several  of  the 
unique  features  of  MultiScheme.  The  purpose  of  these  examples  is  to  in¬ 
troduce  some  novel  ways  of  using  the  tools  provided  by  the  underlying 
MultiScheme  system. 

Section  5.1  concentrates  on  speculative  parallelism  —  using  parallelism 
to  increase  the  speed  of  programs  that  try  multiple  approaches  to  solving 
the  same  problem.  It  shows  a  simple  procedure  (first-value)  which  can 
be  used  as  the  base  for  several  parallel  control  structures,  and  demonstrates 
its  use  in  the  core  of  a  simple  ‘‘expert  system”  (rule-based  interpreter). 
S('cfion  5.2  shows  the  evolution  of  a  serial  program  for  solving  the  n-body 
problem  of  computational  dynamics  into  a  parallel  implementation  by  the 
addition  of  future  constructs.  The  judicious  addition  of  futures,  coupled 
with  measurements  and  a  small  amount  of  rewriting,  ultimately  yields  a 
l)rogram  that  solves  the  8-body  problem  6.6  times  faster  than  the  serial 
version  on  a  16  processor  machine.  Section  5.3  (along  with  some  details  in 
.App<’udix  B)  reviews  tliis  solution  to  the  n-body  problem  and  provides  a 
reformulation  using  a  system  similar  to  a  pipelined  computer  architecture 
or  a  flataflow  computation  model.  Section  5.4  describes  how  the  fork/join. 
Uniform  System [56].  and  Q Lambda [23]  parallel  programming  methods  can 
be  ('xi)r<'ss(’d  in  MultiScheme. 

There  is  no  claim  that  these  programs  provide  optimal  solutions  to  the 
initial  problems  a  carefully  crafted  Fortran  program  coui)led  with  an 
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(ilitinii/iiii!;  Fortran  compiler  can  certainly  generate  faster  programs  for  a 
.Jilt  nti.-il  conumter  than  the  current  interpreter  for  MultiScheme  on  a  32- 
mi,  n•e,"-or  Butterfly  computer.  The  programs  presented  here,  however,  do 
>cale  with  the  size  of  the  problem  and  the  number  of  processors.  Fur- 
riicrmore.  there  is  reason  to  believe  that  an  optimizing  compiler  for  Multi- 
Schemi'  can  be  built  iind  that  the  resulting  compiled  programs  will  both 
p<  rform  well  and  (“.xploit  the  jtarallelism  sirecified  in  the  program'. 


5.1  Speculative  Computation 

There  are  three  imptutant  ways  to  use  multiple  processors  of  a  given  type 
to  -olve  a  ])roblem  faster  than  it  would  be  solved  with  a  single  processor  of 
the  sjune  typce 

•  -Mtmy  problems  can  Ix'  divided  into  loosely  coupled  smaller  problems 
so  that  combining  the  solutions  to  the  smaller  problems  produces  a 
solution  to  the  original  problem.  If  the  coupling  is  sufficiently  loose, 
the  -m.aller  [troldems  can  each  be  assigned  to  a  separate  processor 
and  the  time  reqtiired  to  .solve  them  can  be  overlapped. 

•  In  ;i  proliability-ba.sed  program  (such  as  a  Monte-Carlo  method),  per¬ 
forming  the  same  operation  more  often  in  a  given  amount  of  time 
leads  to  an  improved  answer.  So  for  a  desired  precision,  using  more 
jUDcessois  will  Ic.ad  to  an  answer  in  a  shorter  period  of  time. 

•  If  many  different  methods  are  known  for  solving  a  single  problem 
etich  processor  can  pursue  a  different  approach  to  the  problem.  If  we 
are  looking  for  any  solution  to  the  problem,  then  we  can  select  the 
solution  fouiul  !)}•  tlu'  first  processor  to  announce  success. 

This  third  approach  is  known  as  speculative  parallelism.  Using  the  other 
two  ai)jjroaches  w<'  can  hope  that  ii  processors  would  run  a  program  n 
times  faster  than  one  processor.  By  contrast,  in  speculative  comput<'itions 

■  ria  f.a  ail  eptmilziiig  compiler  is  more  than  sheer  speculation.  An  optimizing 

■<aiipiler  ha  S'  h,  nie':S7]  ha.s  already  heen  implemented.  Work  on  a  similar  oominler  for 
Ml  1  S'-heme  i>  underway.  N'l-edless  In  say.  since  MultiScheme  forms  the  heart  of  the  HB.N 
Hutterlly  I.isji  j, re, duct.  BB-N  i-s  pursuing  compilation  technology  in  the  parallel-processor 
d,-;ii  iin  wall  c, iiisiderahle  vigor 
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we  hope  to  have  n  processors  run  as  fast  as  the  one  processor  that  always 
picks  the  best  method  to  solve  the  problem  at  hand.  Clearly,  this  approach 
is  justified  only  if  there  is  no  simple  way  of  deciding,  in  advance,  which 
approach  works  best  on  the  particular  problem  at  hand. 

5.1.1  Parallel  Control  in  a  Rule  Interpreter 

Speculative  parallelism  may  prove  useful  in  a  variety  of  cases.  Many  “AI 
search”  problems  can  be  conveniently  expressed  using  speculation;  in  fact, 
this  was  one  factor  motivating  the  design  of  the  Connection  Machine[32]. 
The  example  chosen  here,  a  rule  interpreter  that  forms  the  core  of  an  expert 
system,  exhibits  one  such  search.  The  purpose  of  the  example  is  to  demon¬ 
strate  how  speculative  computations  can  be  expressed  in  MultiScheme.  The 
example  does  not  address  other  important  issues  such  as  the  ability  to  ana¬ 
lyze  the  rule  base  to  find  a  better  control  structure  than  the  one  described 
here. 

The  example  is  based  on  a  program  originally  developed  for  teaching 
an  introductory  undergraduate  course  in  signal  processing[6].  It  is  a  rule- 
based  expression  simplifier  supplying  enough  power  to  handle  a  set  of  rules 
for  simple  symbolic  algebra.  Like  most  rule  interpreters  it  consists  of  a  pat¬ 
tern  matcher,  a  rule  firing  mechanism,  and  a  control  structure  component. 
Both  the  pattern  matcher  and  the  control  structure  can  utilize  specula¬ 
tive  parallelism,  but  for  simplicity  this  example  deals  only  with  the  control 
structure. 

The  rule  interpreter  has  an  inner  loop  that  takes  an  expression  and 
tries  to  match  it  against  all  of  the  rules.  If  a  rule  matches  the  expression, 
the  rule  firing  mechanism  is  invoked  to  produce  a  modified  (presumably 
simpler)  expression.  If  no  rule  matches,  then  the  expression  is  already 
in  its  simplest  form.  The  problem  we  arc  attempting  to  solve,  then,  is 
converting  the  sequential  control  structure  of  “multiple  solution  techniques 
augmented  with  a  default  result”  into  a  corresponding  specvilative  (parallel) 
control  structure.  In  doing  so.  we  will  assume  that  rules  may  be  fired  in  any 
order  that  terminates  and  will  still  arrive  at  a  correct  final  result  -  the  only 
restriction  is  that  we  will  not  “give  up”  on  simplifying  an  expression  until 
all  rules  have  failed  to  simplify  it.  This  assumption  is  not  valid  in  general 
for  rule  sets  and  poses  an  important  practical  limitation  to  the  solution 
described  here. 
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MultiSclit'iiK'  j:)r()vi(lt's  two  forms  of  support  for  speculative  ]>rogram- 
mini!;,  and  both  are  used  iu  solving  this  problem:  the  procedure  disjoin 
introdviceil  in  Section  2.5,  and  the  garbage  collection  of  useh'ss  tasks  dis- 
cussi'd  in  Section  3.5.  The  former  is  used  as  the  most  primitive  operation 
tor  reijnesting  sja'culative  computation  it  takes  a  set  of  placelurldt'is 
aiul  returns  a  placehoUler  that  eventually  receives  the  value  of  the  first  one 
to  be  given  a  value.  The  latter  is  invoked  implicitly  rather  than  explic¬ 
itly,  ;ind  serves  to  free  up  processors  that  arc  pursuing  approaches  to  an 
alreai.ly  solved  problem. 

Before  proceeding,  a  little  utility  procedure  %vill  come  in  handy.  The 
procetlure  disjoin  is  a  nice  primitive  mechanism,  but  for  this  example  a 
slightly  different  interface  will  prove  more  convenient.  Rather  than  receive 
a  placeholder  for  the  first  value  to  be  computed,  we  will  want  to  receive 
th('  value  itself.  Furthermore,  we  will  be  creating  a  list  of  placeholders 
se])arately  from  the  call  to  disjoin  so  we  need  a  procedure  that  accepts  a 
list  of  placeholders  (disjoin  expects  the  placeholders  as  arguments).  This 
is  easily  expressed; 

(define  (await-first  list-of -placeholders) 

(touch  ;  Wait  for  a  value  to  appear 

(apply  disjoin  ;  "Spread"  the  arguments 

list-of -placeholders) ) ) 

With  only  this  to  help  us,  it  is  slightly  tricky  to  express  the  notion  of  a 
default  value  to  be  used  only  when  all  other  approaches  fail.  If  we  merely 
use  disjoin  to  try  all  of  the  approaches  we  risk  getting  a  premature  answer 
from  th('  default  method,  causing  us  to  abandon  the  other  techniques.  On 
the  other  hand,  if  we  don’t  include  the  default  value  in  the  set  of  methods 
tried,  we  will  never  succeed  in  simplifying  an  expression  since  we  won't  be 
able  to  siiu])lify  the  most  primitive  components.  The  gist  of  the  solution  is 
to  j)rovide  an  explicit  indication  that  a  rule  has  failed.  Then,  rather  than 
jvist  wait  for  disjoin  to  return  an  answer,  we  look  at  the  answer  we  get 
b;ick.  If  it  is  a  failure  indication  we  wait  again,  until  either  a  successful 
result  is  returned  or  no  other  rules  are  available  only  in  this  last  case  do 
we  supply  the  default  value. 

To  implement  this  idea,  we  begin  by  introducing  the  special  value  used 
to  represent  the  failure  of  a  rule  (failure-tag),  and  a  procedure  to  test 
for  a  not-yet-failed  placeholder: 

(define  failure-tag  (list  ’failure)) 
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(define  (not-f ailed  object) 

(or  (placeholder?  object)  ;  No  value  yet 

(not  (eq?  object  failure-tag))))  ;  Value,  not  failure 

Using  these  it  is  easy  enough  to  write  the  procedure  that  waits  for  a 
non-failure  answer  or  returns  the  default  value; 

(define  (first-value  placeholders  default-value) 

(if  (null?  placeholders) 

default-value  ;  No  alternative 

(let  ((result  (await-first  placeholders))) 

(if  (eq?  failure-tag  result) 

(first-value  ;  Failed:  try  again 

(filter  not-failed  placeholders)  def ault-value) 
result))))  ;  Success 

This  mechanism  is  sufficient  to  solve  our  problem^.  The  rule  system 
discussed  in  [6]  takes  a  set  of  rules  and  produces  a  procedure  for  simplifying 
expressions  based  on  these  rules.  It  consists  of  three  major  procedures, 
corresponding  to  the  pattern  matcher,  rule  firing  mechanism,  and  control 
structure.  For  our  purposes,  the  first  two  components  can  be  treated  as 
"black  boxes”  whose  internal  operation  is  not  relevant  to  us.  The  pattern 
matcher  and  rule  firing  components  have  the  following  interfaces  (these 
differ  slightly  from  the  version  in  [6]): 

(matcher  rule  expression  fail  succeed) 

Matches  the  rule  against  the  expression.  If  there  is  no  way  for 
them  to  match,  fail  is  called  with  no  arguments.  Otherwise  suc¬ 
ceed  is  called  with  a  dictionary  (containing  the  values  assigned  by 
the  successful  match  to  variables  in  the  pattern)  and  a  procedure 
that  can  be  used  to  resume  the  matching  process  if  the  values  in  the 
dictionary  are  unacceptable  (fire,  below,  is  the  base  for  a  typical 
success  procedure  as  shown  in  Figure  5.1). 
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(fire  rule  dictionary  simplifier) 

Fires  the  specified  rule  using  dictionary  to  guide  construction  of 
a  new  expression.  The  resulting  new  expression  is  simplified  using 
simplifier  and  returned. 

The  control  component  —  the  lone  component  we  are  setting  out  to 
modify  -  consists  of  the  routine  simplifier  shown  in  Figure  5.1.  This 
simplifies  any  sub-expressions,  then  tries  the  rules  sequentially  until  one 
fires.  The  firing  process  then  creates  a  replacement  expression  and  (re¬ 
cursively)  simplifies  it.  The  only  part  of  simplifier  that  we  will  need  to 
change  is  scan,  which  in  the  original  version  (reproduced  below)  sequen¬ 
tially  scans  the  rules; 

(define  (scan  rules) 

(if  (null?  rules) 
exp 

(match-rule  (car  rules) 

(lambda  ()  (scan  (edr  rules)))))) 

In  this  sequential  version,  scan  tries  the  first  rule  —  using  match-rule 
—  and  specifies  that  if  this  rule  fails  to  fire  the  process  should  repeat 
with  the  remaining  rules.  Instead  of  this,  we  want  to  use  the  first-value 
procedure  we  defined  above  by  creating  a  task  for  each  of  the  rules.  In  order 
to  do  this,  we  must  arrange  matters  so  that  when  a  rule  fails  to  match  it 
causes  the  task  that  is  running  to  return  the  failure  signal: 

(define  (scan  rules) 

(define  workers  ;  List  of  placeholders 

(map  (lambda  (rule)  ;  ...one  per  rule 

(FUTURE  (match-rule  rule 

(lambda  ()  failure-tag)))) 

rules) ) 

(first-value  workers  exp)) 

Introducing  this  form  of  speculative  parallelism  into  the  rule  interpreter 
was  quite  simple.  But  the  same  technique  allows  more  complicated  sequen¬ 
tial  control  structures  to  be  converted  to  parallel  speculation  as  well.  By 
only  a  small  change  to  first -value  it  can  be  converted  into  the  basic  sup¬ 
port  needed  fiu  handling  any  number  of  priority-based  solution  methods 

instead  of  returning  a  specific  value,  it  can  call  a  procedure  that  starts 


(define  (simplifier  the-rules) 

(define  simplify-exp  ;  (1) 

(memorize 
(lambda  (exp) 

(try-rules  (if  (compound?  exp) 

(map  simplify-exp  exp) 
exp))))) 

(define  (try-rules  exp)  ;  (2) 

(define  (match-rule  rule  rule-fail)  ;  (3) 

(matcher  rule  exp  rule-fail 

(lambda  (dictionary  keep-matching) 

(fire  rule  dictionary  simplify-exp)))) 

(define  (scan  rules)  ;  (4) 

(if  (null?  rules) 
exp 

(match-rule  (car  rules) 

(lambda  ()  (scan  (cdr  rules)))))) 

(scan  the-rules)) 
simplify-exp) 

Notes: 

1.  Simplify-exp  simplifies  all  the  components  of  an  expression,  then  tries  all  of  the 
rules  on  the  result.  Memorize  implements  the  standard  dynamic  programming  trick 
of  consulting  a  table  of  known  simplifications  before  attempting  the  recursive  tree 
walk  shown  here. 

2.  Try-rules  tries  to  match  and  fire  each  rule  against  the  incoming  expression. 

3.  Match-rule  tries  to  match  and  fire  a  single  rule  against  the  incoming  expression. 

4.  Scam  sequentially  tries  to  match  and  fire  each  rule  against  the  incoming  expression. 

Figure  5.1;  Recursive  Control  Structure 
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the  next  lower  priority  set  of  methods  and  calls  first-value  to  wait  for 
all  of  those  methods  to  complete,  and  so  forth.  Since  there  need  be  no 
restriction  on  the  mechanism  used  to  generate  the  next  set  of  methods  to 
be  tried,  quite  complicated  systems  of  dependencies  can  be  expressed  using 
the  same  general  technique. 

5.1.2  Speculative  Computation:  Summary 

.4s  an  e.xample,  this  rule  system  was  run  using  two  sets  of  rules  —  one 
for  symbolic  algebra  and  one  for  derivatives  —  and  these  two  rule  sets 
were  passed  as  arguments  to  the  simplifier  procedure  to  create  two  sim¬ 
plification  functions.  These  two  functions  were  themselv'es  composed  to 
produce  one  large  procedure,  psimp,  that  first  simplifies  its  input  using  the 
derivative  rules  and  then  simplifies  the  result  using  the  algebra  rules.  The 
algebraic  rule  set  consisted  of  twenty  rules  including  constant  elimination, 
commutativity,  distributivity,  and  conversion  of  products  into  exponents. 
The  derivative  rule  set  had  eight  rules. 

The  psimp  procedure  was  used  to  find  the  derivative  of  a  standard 
cubic  equation  {ax^  +  bx^  +  cx  +  d  with  respect  to  x).  Using  ten  processors, 
the  deriv^ative  calculation  required  half  the  time  of  the  same  calculation 
on  a  single  processor.  One  property  of  the  rule  sets  (unfortunate  for  this 
example)  is  that  a  single  expression  rarely  matches  the  pattern  of  more 
than  one  rule.  As  a  result,  most  tasks  terminate  very  quickly  by  failing  to 
match,  so  the  garbage  collection  of  tasks  played  no  noticeable  role.  The 
ability  of  the  garbage  collector  to  remove  useless  threads  of  computation 
remains  important  in  general,  although  this  rule  set  and  parallel  control 
structure  do  not  show  it  to  advantage. 

The  jiarticular  choice  of  speculative  control  structure  used  here  is  only 
one  of  a  range  that  deserves  more  exploration.  An  interesting  property  of 
this  structure  is  that  it  allows  the  parallel  system  to  arrive  at  answers  even 
in  cases  where  a  sequential  implementation  would  fail  because  of  an  infinite 
loop  in  the  rule  set.  If,  for  example,  we  include  commutativity  as  one  of 
ndes  of  algeVjra  it  is  essential  in  the  serial  system  that  this  rule  be  chosen 
only  once  for  any  given  expression  otherwise  we  might  start  with  a  +  h. 
conv(’rt  it  to  b  +  n,  then  convert  it  back  to  <i  -f-  b  and  so  forth*.  By  contrast. 


■’Tlie  rule  lia.se  used  in  tlie  .serial  program  of  [6]  solves  this  proMeni  hy  the  use  of 
rest  riri  ifius  on  the  applicability  of  the  rule.  A  lexical  ordering  on  expressions  is  exploited 
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the  parallel  control  structure  here  will  terminate  even  in  this  case  —  and 
then  the  garbage  collector  would  remove  the  remaining  tasks.  Without 
performing  such  an  experiment,  however,  it  is  not  clear  just  how  well  such 
a  rule  set  would  perform. 

This  experiment,  while  demonstrating  a  factor  of  2  performance  im¬ 
provement,  leaves  open  a  number  of  questions  related  to  the  use  of  specu¬ 
lative  parallelism  in  rule-based  systems.  But,  2is  mentioned  at  the  outset, 
answering  these  questions  was  not  the  purpose  of  the  example.  Far  more 
importantly,  this  example  has  shown  that  the  underlying  disj  oin  operation 
does  provide  the  starting  point  for  building  systems  that  exploit  parallelism 
to  explore  alternative  methods  for  solving  a  single  problem.  Using  this  op¬ 
eration  it  was  a  straightforward  task  to  modify  the  serial  control  structure 
of  an  existing  system  into  one  more  suited  for  a  parallel  processor. 

5.2  Object  Interaction  Simulator 

One  class  of  problem  in  which  parallel  computation  can  be  expected  to 
provide  significant  performance  improvement  is  the  simulation  of  systems 
composed  of  interacting  components.  The  plan  is  to  utilize  the  multiple 
computation  units  so  that  each  simulates  a  single  object  or  group  of  objects, 
periodically  updating  its  own  state  based  on  the  state  of  all  of  the  other 
objects.  This  kind  of  simulation  is  quite  common,  and  has  been  a  target 
of  investigation  in  the  parallel  computing  community  for  some  time.  One 
classic  example  of  this  kind  of  problem  is  the  “n-body  problem”  of  classical 
physics: 

Given:  The  mass  (/u,),  initial  position  (f*,(0)),  and  initial  velocity  (i\(0)) 
of  a  group  of  n  particles. 

Given:  The  force  law  (F(f*,,xj))  governing  the  interaction  of  a  pair  of 
particles. 

Predict:  The  position  and  velocity  (i\(t))  of  the  particles  into  the 

future  (possibly  ignoring  collisions). 

The  problem  is  easily  formalized,  but  the  computational  task  for  even  rela¬ 
tively  small  systems  is  immen.se.  Becatise  the  interparticle  force  calculations 

to  guarantee  that  the  rule  i.s  fired  only  once  for  a  given  expression 
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rp  steps  a  serial  processor  recpiires  time  j)rop«)rtioual  to  tP.  Since 
tiie  prolileiii  inliereauly  divitles  into  n  iiulepentlent  p.arts  (one  per  particle), 
it  i>  I'lear  that  a  program  can  he  coded  that  will  solve  the  prohh'in  in  time 
proj^ortional  to  n  if  tli(>re  are  n  proces.sors  available. 

The  description  and  analysis  of  this  single  examph'  is  quite  long  (it 
sj)ans  both  Sections  5.2  and  5.3)  and  provides  information  at  a  number  of 
le\'els.  At  one  level  it  provides  a  largely  acc\irate  chronology  of  the  con- 
v('rsioii  of  a  working  serial  program  into  a  parallel  program  that  achieve.s 
the  desiretl  sp('ed  increase.  At  anotlu'r  h'vel.  the  measurements  that  ino- 
ti\ateil  each  step  in  the  process  are  thems<dves  aided  by  modifications  to 
the  MultiScherne  schedider  that  provided  the  data  needed  to  graphically 
I>resent  the  parallelism  j)rofile  of  the  program,  illustrating  the  value  of  a 
tiexiljle  scheduler  implementation  in  MultiScherne.  At  yet  another  leved, 
the  clnuigt's  from  each  version  of  the  program  to  the  next  demonstrat('  uses 
of  the  constructs  descrilx'd  earlier.  And  finally,  the  last  transformation 
(described  in  Section  5.3)  of  the  program  demonstrates  a  technicjue  that 
h;is  long  been  the  strong  point  of  Lisp  languages;  the  ability  to  constrtict  a 
system  that  i)ro\-ides  the  external  behavior  of  a  completely  different  model 
of  computation  ba.sed  on  primitives  that  exist  within  (in  this  case)  Multi- 
ScIk'H'C. 

5.2.1  Simple  Sequential  Implementation 

To  begin  imi)l(’inenting  a  parallel  program  that  solves  the  n-body  prob¬ 
lem,  let  us  first  study  an  elegant  sequential  solution.  This  program  is 
tidapted  from  ;i  version  written  by  Sussrnan,  ba.sed  c^n  his  work  on  orbital 
inechani<’s;S].  Tlie  i)rogram  is  structured  around  the  notion  of  a  system 
of  particles  and  ojx-rations  on  such  systems.  It  operates  by  constructing 
differential  systems  rei)res<'nting  changes  to  a  system  over  some  small  tinu' 
inter\  a!  .Inst  as  a  system  consists  of  j)articles.  a  diffr'rential  system  consists 
'  )t  differential  particles  whose  state  is  comjruted  based  on  the  state  of  the  orig 
inal  paifich's  and  a  irarticle  interaction  (force)  law.  Using  a  Runge-Kutta 
uir'  sriaMon  tei  hnique  to  combine  the  initial  system  with  related  differen- 
Mal  -y^rciM'-  the  ]ii()gram  irrcdicfs  flie  ix)sition  and  velocity  of  the  ])arficles 
'.mo  tho  fiitiii'-.  The  solution  makes  use  of  a  library  of  gem'ral-purpose 
!;i;iri,cni;jtii';il  routines,  shown  in  Figur*’ 5.2. 

.\  '■\  -reni  consists  of  a  siniulateil  time,  .and  the  set  (list)  of  partich's  in 
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Name 

Description 

(group  op  11) 

Re-groups  objects  from  11,  a  list  of  sets  of  objects, 
into  sets  based  on  a  selection  criterion  (op).  Each 
output  set  consists  of  those  input  objects  for  which 
op  returns  the  same  value. 

(reduce  op  vals) 

Combines  all  of  the  vals  using  the  associative  op¬ 
erator  op. 

Scalar  Operations 

cube,  square  and  sqrt  (square  root). 

(scale  n) 

Returns  the  procedure  that  multiplies  a  vector  by 
n  (a  number). 

(scale-by  n  v) 

Scale  vector  v  by  amount  n.  More  convenient  to 
write  than  ((scale  n)  v). 

W'ctor  Operations 

add  and  sub. 

zero-vector 

A  pre-computed  three  dimensional  vector  of  zeros. 

Figure  5.2:  A  Simple  Mathematics  Library 
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the  system.  The  particles  have  their  own  state  information,  represented  for 
simplicity  as  a  vector  of  quantities.  Each  particle  has  a  name,  primarily 
for  documentation  purpo.ses.  It  also  has  a  mass  (a  scalar  quantity)  and 
two  vectors  representing  its  position  and  velocity.  Particles  are  constructed 
using  the  procedure  make-particle,  systems  by  make-system.  Selectors 
are  time  and  particles  for  a  system,  and  name,  mass,  position  and 
velocity  for  particles. 

There  are  two  operations  that  can  be  performed  on  particles.  Scale- 
Particle  (used  to  construct  differential  .systems)  multiplies  the  state  infor¬ 
mation  of  a  particle  by  a  scalar  factor  (representing  the  time  step).  Add- 
Particles  adds  together  the  state  information  from  a  number  of  particles 
to  jiroduce  the  state  information  for  a  new  particle.  See  Figure  5.3. 

(define  (scale-particle  factor) 

(define  factor*  (scale  factor))  ;  vector  scale 

(lambda  (particle) 

(make-particle 

(name  particle)  ;  name 

(*  factor  (mass  particle))  ;  mass 

(factor*  (position  particle))  ;  position 

(factor*  (velocity  particle)))))  ;  velocity 

(define  (add-particles  .  particles) 

(make-particle 

(name  (car  particles))  ;  name  (same  for  all) 

(reduce  +  (map  mass  particles))  ;  mass 

(reduce  add  (map  position  particles))  ;  position 

(reduce  add  (map  velocity  particles))))  ;  velocity 

Figure  5.3;  Operations  on  Particles  (serial) 

Finally,  there  are  two  operations  on  systems  of  particles.  These  corre- 
''jamd  to  the  j)article  operations;  scale-system  multiplies  a  system  by  a 
constant  factor  (time  step),  and  add-systems  sums  a  number  of  systems 
to  i)roduce  a  resultant  system.  See  Figure  5.4. 

The  most  complicated  st<>p  in  our  simulation  is  to  write  a  procedure 


(define  (scale-system  scale-factor) 

(define  sp  (scale-particle  scale-factor)) 

(lambda  (system) 

(make-system  (*  scale-factor  (time  system)) 

(map  sp  (particles  system))))) 

(define  (add-systems  .  systems) 

(make-system  (reduce  +  (map  time  systems)) 

(map  (lambda  (bunch)  (reduce  add-particles  bunch)) 
(group  name  (map  particles  systems))))) 

Figure  5.4:  Operations  on  Systems  of  Particles  (serial) 

that  will  construct  a  “differential  system”  from  an  initial  system  and  a  par¬ 
ticle  interaction  force  law  (see  Figure  5.5).  A  differential  system  consists 
of  a  differential  of  time  {dt/dt,  clearly  always  1),  and  a  set  of  differential 
particles.  Just  as  a  particle  consists  of  a  mass,  position,  and  velocity,  a 
differential  particle  consists  of  a  differential  of  mass  {dmfdt,  always  0  since 
there  is  assumed  to  be  no  loss  of  mass  over  time),  a  differential  of  position 
(dxfdt,  which  is  of  course  the  velocity  of  the  original  particle),  and  a  differ¬ 
ential  of  velocity  {dv/dt,  or  acceleration).  This  last  component  is  the  core 
of  the  computation.  Acceleration  is  just  the  sum  of  the  inter-particle  forces 
acting  on  a  given  particle  (divided  by  the  mass  of  the  particle),  computed 
using  the  specified  force-law. 

Given  this  framework,  we  can  describe  the  overall  n-body  computation. 
All  that  needs  to  be  done  is  to  select  an  integration  procedure  sufficiently 
accurate  for  our  purposes.  One  could,  for  example,  merely  use  the  dif¬ 
ferential  system  to  produce  a  linear  prediction  of  the  next  state.  This 
would  suffice  for  either  a  very  small  value  of  dt  or  a  low  accuracy  calcula¬ 
tion.  There  are  many  more  sophisticated  (and  accurate)  techniques  that 
have  been  developed  for  numerical  integration,  and  for  this  example  a  4'^ 
order  Runge-Kutta  integrator"'  was  used.  The  code  for  this  specific  integra- 

■'A  Runge-Kutta  integrator  works  by  using  interpolation  of  the  system  based  on  deriva¬ 
tives  of  the  system  state  up  to  some  chosen  number.  A  ith  order  integrator,  thus,  uses  the 
first  through  fourth  derivatives  of  the  system.  It  is  a  generalization  of  the  more  familiar 
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(define  (particle-force  force-law) 

(define  (accelerations  bodies)  ;  (1) 

(define  (my-acceleration  me)  ;  (2) 

(reduce  add 

(map  (lambda  (other)  (force-law  me  other)) 
bodies) ) ) 

(map  my-acceleration  bodies)) 

(lambda  (system-state)  ;  (3) 

(make-system  1  ;  dt/dt 

(map  (lambda  (particle  acceleration) 

(make-particle 
(name  particle) 


0  :  dm/dt 

(velocity  particle)  ;  dx/dt 

acceleration))  ;  dv/dt 

(particles  system-state) 

(accelerations  (particles  system-state)))))) 


(define  (gravitation  me  it)  ;  (4) 

(if  (eq?  me  it) 
zero-vector 

(let  ((dx  (sub  (position  me)  (position  it)))) 

(let  ((rcube  (cube  (sqrt  (reduce  +  (map  square  dx)))))) 
(scale-by  (/  (*  -G  (mass  it))  rcube)  dx))))) 


I 


i 


I 


! 


! 


Notes: 

1.  Return  a  list  of  the  accelerations  of  all  n  particles. 

2.  Calculate  the  acceleration  of  one  particle. 

3.  Procedure  returned  from  Particle-Force,  that  takes  a  system  at  time  t  and  pro¬ 
duces  the  differential  system  dt.  See  text  beginning  on  page  129. 

4  A  sample  force  la\v  returning  the  acceleration  of  me  as  a  result  of  gravitational 
interaction  with  it. 

Figure  5.5:  Differential  System  Generator 
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tiou  technique  (runge-kutta-4)  along  with  the  procedure  that  produces  a 
stream  of  future  values  of  an  initial  system  (integrate-system)  are  shown 
in  Figure  5.6. 

The  serial  program  is  now  complete.  It  can  be  tested  out  by  creating 
a  system  of  particles  and  predicting  the  state  of  the  system  for  three  time 
steps.  Using  the  9-particle  solar  system^  as  an  example,  we  have  (time  in 
days,  distance  in  AU,  mass  in  units  of  the  mass  of  the  sun,  center  of  mass 
coordinate  system): 


(define  sun 

(make-particle 

’sun+mercury  (+  1  (/  1  6000000))  ;  name  and  mass 

(-4.09433e-3  -5.62963e-3  -2.2635e-3)  ;  position 

(6.666a-6  -5.75161e-6  -2.63530-6)))  ;  velocity 


(define  system  (make-system  0  (list  sun+mercury  ...))) 
(define  system-stream  (integrate-system  system  0.5)) 
(nth-stream  system-stream  3) 

(1.5  :  time 

(SUN+MERCURY  ;  name 


1 .  ;  mass 

(-4.08432a-3  -5.63825e-3  -2.26745e-3)  ;  position 

(  6.67693e-6  -5.73931e-6  -2.63031e-6))))  ;  velocity 

...)) 


5.2.2  Parallel  Particle  Interaction 

The  system  as  described  above  works  well,  and  it  is  time  to  speed  it  up  with 
the  judicious  addition  of  future.  Because  the  computation  is  asymptoti¬ 
cally  dominated  by  the  time  rccjuired  to  compute  particle  interactions,  it 
seems  logical  to  rewrite  the  particle-force  procedure  of  Figure  5.5.  This 
is  easily  done,  by  changing  only  the  procedure  my-acceleration  within 
particle-force; 

(Idif  l(‘s.s  accurate)  trapezoidal  teclini<iue  for  integration. 

^'Tliis  treats  the  sun  and  .Mercury  as  a  single  particle,  a. standard  simplification  used  in 
■a.st  ro[)hysic.s. 


*  ^  * 


(define  (runge-kutta-4  f  dt)  ;  (1) 

(define  dt*  (scale-system  dt)) 

(define  2*  (scale-system  2)) 

(define  1/2*  (scale-system  (/  1  2))) 

(define  1/6*  (scale-system  (/  1  6))) 

(lambda  (system)  ;  (2) 

(define  kO  (dt*  (f  system))) 

(define  kl  (dt*  (f  (add-systems  system  (1/2*  kO))))) 

(define  k2  (dt*  (f  (add-systems  system  (1/2*  kl))))) 

(define  k3  (dt*  (f  (add-systems  system  k2)))) 

(add-systems  system 

(1/6*  (add-systems  kO  (2*  kl)  (2*  k2)  k3))))) 

(define  (integrate-system  initial- state  dt)  ;  (3) 

(let  ((integrator  (runge-kutta-4  (particle-force  gravitation)  dt)/) 
(define  (next  state) 

(cons  state 

(delay  (next  (integrator  state))))) 

(next  initial-state))) 

Nfjtos; 

1  [’rodiicf'  a  [iroccduro  that  takes  an  input  system  at  time  t  and  produces  a  system 
at  time  t  +  dt  using  a  4'^  order  Runge-Kutta  integrator. 

2.  rtiis  is  llie  actual  integration  procedure  returned  from  Runge-Kutta-4. 

3  Produce  a  stream  of  values  representing  the  state  of  the  system  beginning  with  the 
initial-state  ai\d  separated  by  intervals  of  dt. 


Figtire  5.G:  Integration  Procedure  (serial) 
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(define  (my-acceleration  me) 

(FUTURE 

(reduce  add  (map  (lambda  (other)  (force-law  me  other)) 
bodies)))) 

Now,  in  order  to  perform  the  inter-particle  force  calculations  the  pro¬ 
gram  creates  a  task  for  each  particle.  That  task  calculates  the  acceleration 
of  its  associated  particle  based  on  the  state  information  of  all  of  the  par¬ 
ticles.  Installing  this  change  and  running  the  program  yields  precisely  the 
same  results  as  before,  but  faster. 

Our  initial  expectation,  of  course,  was  that  for  a  9-particle  system  we 
should  see  a  factor  of  9  improvement  (since  the  experiment  was  run  on  a 
machine  with  more  than  9  processors).  Unfortunately,  the  program  runs 
only  a  factor  of  1.8  faster  than  the  original  —  the  kind  of  result  often 
encountered  when  adding  parallelism  to  a  serial  program.  At  this  point, 
the  project  shifts  into  a  “performance  debugging”  phase,  not  unlike  the 
ordinary  debugging  phase  of  a  program.  At  this  stage  the  ability  to  visualize 
the  parallelism  being  attained  by  the  program  is  essential. 

Fortunately,  MultiScheme  running  on  the  Butterfly  computer  had  al¬ 
ready  been  augmented  by  Seth  Steinberg  (of  the  BBN  Butterfly  Lisp  pro¬ 
ject  )  to  provide  a  good  deal  of  help  with  this  task.  He  modified  the  scheduler 
described  in  Chapter  4  to  send  a  monitoring  packet  out  whenever  a  task  is 
created,  suspended,  or  completes  its  processing.  These  packets  are  collected 
and  can  be  displayed  (unfortunately  only  after  a  run  is  completed)  using 
a  set  of  tools  that  are  part  of  the  “Butterfly  Lisp  User  Interface”  (BLUI) 
developed  by  Laura  Bagnall  of  the  BBN  project.  A  glance  at  these  displays 
(see  Figure  5.7  and  the  accompanying  notes)  reveals  a  consistent  pattern: 

•  A  short  stretch  of  3-fold  parallel  computation. 

•  A  medium  length  stretch  of  serial  computation. 

•  The  previous  two  items  repeat  a  total  of  four  times. 


.4  long  period  of  serial  computation. 


Clearly,  the  9-fold  parallel  unit  that  repeats  four  times  arises  from  the 
four  calls  to  f  within  the  procedure  runge-kutta-4.  The  medium  length 
serial  computation  between  them  corresponds  to  the  work  of  scaling  and 
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Key  to  Performance  Diagrams: 

Performance  diagrams  consist  of  two  regions,  with  a  common  time  axis  running  from  left  to 
right.  The  lower  region  is  a  graph  of  processor  activity  showing  the  number  of  processors 
actually  engaged  in  running  tasks.  The  upper  portion  is  a  detailed  accounting  of  all  tasks 
in  the  system  during  the  time  of  interest.  Each  task  is  represented  by  a  horizontal  bar 
which,  in  enlarged  drawings,  will  be  seen  to  consist  of  segments  in  three  different  patterns: 
fully  black  indicating  a  running  task,  dark  grey  indicating  a  task  that  is  ready  to  run  but 
for  which  no  processor  is  available,  and  white  (very  light  grey)  indicating  a  task  waiting 
for  a  placeholder’s  value  to  become  available.  The  vertical  arrows  indicate  either  a  task 
being  spawned  (arrow  pointing  down  from  the  parent,  which  is  black,  to  the  child),  a  task 
waiting  for  the  value  of  another’s  goal  placeholder  (arrow  from  the  task  that  is  waiting 
—  and  thus  changing  from  black  to  white  —  to  the  task  that  will  deliver  the  value),  or  a 
dstanlas !  operation  (typically  terminating  a  horizontal  bar). 

The  range  of  times  shown  on  these  performance  diagrams  is  not  the  same  on  every  diagram. 


Figure  5.7:  Paxetllelism  in  Particle  Force 
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adding  the  resulting  system.  The  final  long  serial  computation  corresponds 
to  the  final  call  to  add-systems. 

These  results  were  not  overly  surprising,  although  there  seemed  to  be 
no  reason  for  the  final  call  to  add-systems  to  be  so  very  much  (more  than 
a  factor  of  4)  longer  than  the  earlier  calls.  Re-examining  the  procedure  in 
Figure  5.4  shows  that  one  possible  problem  is  in  the  group  procedure  of 
the  math  library.  Indeed,  examining  that  code  (not  included  here)  reveals 
that  it  employs  an  algorithm  with  a  worst  case  performance.  A  minimal 
.'peed  increase  can  be  found  by  rewriting  group  to  itself  operate  in  parallel, 
but  a  more  dramatic  increase  results  from  understanding  what  purpose  it 
is  serving. 

In  general,  when  two  systems  of  particles  are  to  be  added  together,  there 
is  no  guarantee  that  the  system  contains  precisely  the  same  particles.  Fur¬ 
thermore,  we  have  implemented  systems  using  unsorted  lists  to  represent 
the  sets  of  particles.  As  a  result,  there  is  no  guarantee  that  even  if  the  sys¬ 
tems  contain  the  same  particles  they  will  appear  in  the  same  order  in  each 
system.  Thus,  a  general  purpose  procedure  for  adding  together  systems 
of  jjarticles  must  use  some  operation  like  group  to  match  corresponding 
particles  in  the  systems.  In  the  problem  at  hand,  however,  the  particles 
in  each  system  are  in  fact  always  both  the  same  and  in  the  same  order. 
Thus,  rewriting  add-systems  with  this  assumption  will  save  considerable 
time  during  the  computation.  At  the  same  time,  we  can  operate  on  each 
set  of  corresponding  particles  in  parallel: 

(define  (add-systems  .  systems) 

(define  (transposer  particle-set) 

(define  (compute  particles)  (reduce  add-particles  particles)) 

(if  (null?  (caar  particle-set)) 

’() 

(cons  (FUTURE  (compute  (map  car  particle-set))) 

(FUTURE  (transposer  (map  cdr  particle-set)))))) 
(make-system  (reduce  +  (map  time  systems)) 

(transposer  (map  particles  systems)))) 

With  this  new  version  of  add-systems  performance  is  significantly  bet¬ 
ter.  i)ut  only  by  a  factor  of  3.7  over  the  original.  (See  Figure  5.8  for  a 
close-vip  of  one  of  the  four  system  calculations.  This  figure  corresponds 
to  the  region  between  the  points  marked  A  and  B  in  Figure  5.7.)  The 
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This  figure  corresponds  to  the  area  between  the  points  marked  A  and  B  in  Figure  5.7. 
S'ae  the  notes  to  Figure  5.7  for  a  general  description  of  performance  diagrams.  In  this  figure, 
the  numbers  visible  in  the  horizontal  bars  show  the  number  of  the  processor  running  the 
task. 

Figure  5.8:  Parallelism  in  Adding  Systems 
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next  advance  comes  from  realizing  that  the  procedure  scale-system  can 
be  improved  by  processing  the  particles  in  parallel: 

(define  (scale-system  scale-factor) 

(define  sp  (scale-particle  scale-factor)) 

(define  (scale-it  x)  (FUTURE  (sp  x))) 

(lambda  (system) 

(meOce-system  («  scale-factor  (time  system)) 

(map  scale-it  (particles  system))))) 


See  the  notes  to  Figure  5.7  for  a  general  description  of  performance  diagrams. 

Figure  5.9:  Parallelism  in  Scaling  Systems 

This  was  not,  perhaps,  obvious  at  first  since  the  work  entmled  in  scaling 
a  particle  seems  small  —  small  enough  that  it  isn’t  likely  to  compensate 
for  the  work  necessary  to  create  a  new  task.  On  reconsideration,  however, 
the  cost  of  the  two  vector  operations  involved  in  scaling  a  particle  (one  for 
position  and  the  other  for  velocity)  is  sufficient  to  overshadow  this  cost. 
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This  figure  is  an  enlargement  of  the  area  between  the  poinU  marked  A  and  B  in  Figure  5.9 
See  the  notes  to  Figure  5.7  for  a  general  description  of  performance  diagrams.  Unlike  the 
earlier  examples,  the  amount  of  parallelism  demanded  exceeded  the  16  processors  available. 
Hence  some  of  the  bars  are  grey,  indicating  queuing  of  tasks  waiting  for  processors. 

Figure  5.10:  Details  of  Scaling  Systems 
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With  this  change  we  begin  our  final  approach  to  the  speed-up  we  had 
originally  expected.  The  figure  now  stands  at  a  G. 6-fold  increase  over  the 
i.)riginal  serial  version.  Further  work  along  the  same  lines  doesn’t  seem 
warranted,  since  examination  of  the  activity  graphs  (see  Figure  5.9  for 
the  total  computation,  and  an  enlargement  in  Figure  5.10)  indicates  no 
significant  areas  of  serial  computation  —  we  are  seeing  the  overhead  of 
creating  the  tasks  on  demand  combined  with  small  amounts  of  truly  serial 
computation  (such  as  the  sequencing  of  execution  of  the  defines  in  runge- 
kutta-4). 

5.2.3  Particle  Interaction:  Summary 

As  we  have  seen,  the  simple  methodology  of  adding  future  constructs  to  a 
straightforward  serial  solution  of  the  n-body  (particle  interaction)  problem 
does,  indeed,  succeed  in  producing  a  faster  program  by  exploiting  the  in¬ 
herent  parallelism.  Unfortunately,  intuition  is  not  a  perfect  guide  to  where 
this  parallelism  can  be  used  to  best  advantage.  A  set  of  tools  for  visualizing 
where  the  parallelism  is  being  exploited  and  (more  importantly)  where  com¬ 
putation  remains  serial  is  almost  essential  to  the  “performance  debugging” 
task. 

But  this  simple  methodology  h£is  some  drawbacks.  This  example  has 
demonstrated  two  of  them.  The  first,  perhaps  a  failing  of  the  author  rather 
than  the  methodology,  comes  from  a  poor  intuition.  Even  knowing  the 
c(jst  of  a  future  and  applying  good  engineering  trade-off  concepts,  the 
initial  parallel  version  missed  a  significant  use  of  parallelism.  This  was 
corrected  only  by  using  the  visualization  tools  to  actually  measure  the  time 
spent  during  the  scale-particle  calculations.  This  kind  of  engineering 
estimation  can  be  inqjroved,  perhaps,  by  the  use  of  automated  program 
analysis  tools.  This  approach  has  been  explored  to  a  limited  degree  by 
Sharon  Gray [25]. 

The  se'cond  drawback,  however,  is  harder  to  correct,  .\ftcr  some  effort  a 
geiH'ial  purpose  utility  procedure  was  found  to  use  a  large  amount  of  serial 
computation  to  solve  a  very  simple  problem.  Just  as  in  a  serial  program,  re¬ 
coding  the  general  algorithm  using  specialized  knowledge  of  the  application 
produces  a  significant  improvement.  Unlike  the  first  drawback,  attempting 
to  automate  this  process  brings  us  dangerously  close  to  the  automatic  pro- 
grainniing  area.  In  fact,  the  effort  to  discover  and  classify  algorithms  that 
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exploit  parallelism  is  a  major  area  of  research  in  the  theory  of  computation. 
While  the  actual  future  construct  itself  has  nothing  whatsoever  to  do  with 
the  difficulty,  its  very  simplicity  encourages  a  style  of  operation  where  this 
kind  of  algorithmic  exploration  is  easily  overlooked. 

5.3  Parallel  Pipelining 

The  solutions  to  the  n-body  problem  described  in  Section  5.2  were  arrived 
at  by  incremental  changes  to  a  working  sequential  solution.  This  section 
takt's  a  different  approach  to  solving  the  same  problem.  First  a  new  general 
framework  for  expressing  parallel  programs  is  created.  Then  this  specific 
problem  is  solved  within  that  framework.  This  section  will  serve  to  high¬ 
light  two  iinj>ortant  points.  First,  one  of  the  most  powerful  aspects  of 
MultiScheme  is  the  case  with  which  alternative  programming  models  can 
be  expressed  and  the  resulting  ability  to  experiment  with  new  techniques. 
Second,  this  section  introduces  some  uses  of  placeholders  and  t^lsks  that 
are  less  evident  than  the  ones  in  earlier  examples. 

The  speed  increase  of  a  factor  of  6-6  out  of  a  possible  9.0  that  was 
achieved  in  Section  5.2  seems  about  the  maximum  we  csui  reach  using  the 
simple  technique  of  adding  futures  to  a  sequential  program.  And  yet  with 
one  final,  less  obvious  change,  we  can  reach  a  speed-up  of  7.9  over  the  serial 
version.  This  final  change  occurs  in  runge-kutta-4.  Notice  that  the  two 
intermediate  constants  kl  and  k2  appear  in  the  final  line  seeded  by  2.  This 
scaling  computation  can  begin  earlier,  providing  some  additional  speed,  by 
mert'ly  introducing  two  new  intermediate  variables: 

(define  (runge-kutta-4  f  dt) 

...  definitions  of  dt*.  2*,  1/2*,  and  1/6*  as  in  Figure  5.6  ... 

(lambda  (y) 

(define  kO  (dt*  (f  y))) 

(define  kl  (dt*  (f  (add-systems  y  (1/2*  kO))))) 

(define  kl*2  (FUTURE  (2*  kl)))  ;  *** 

(define  k2  (dt*  (f  (add-systems  y  (1/2*  kl))))) 

(define  k2*2  (FUTURE  (2*  k2)))  ;  *** 

(define  k3  (dt*  (f  (add-systems  y  k2)))) 

(add-systems  y  (1/6*  (add-systems  kO  kl*2  k2*2  k3)))))  ;  *** 

This  new  way  of  using  parallelism  to  speed  up  the  program  is  reminis¬ 
cent  of  the  arguments  put  forth  in  favor  of  the  dataflow  model  of  comput- 
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ing.  Ill  fact,  this  problem  seems  ideally  suited  for  a  data  flow  computation 
model.  With  this  idea  in  mind,  we  can  explore  an  alternative  implementa¬ 
tion  technique  for  this  jiroblem  that  maintains  the  same  modularity  in  the 
solution. 

The  new  solution  to  the  n-body  problem  takes  a  sort  of  vector  pipeline 
approach  to  the  problem.  Imagine  a  pipeline  that  consists  of  five  kinds  of 
luiilding  blocks: 

1.  Straight  segments  where  computations  on  individual  elements  of  the 
vector  occur. 

2.  Forks  where  the  vector  proceeds  in  multiple  directions  simultaneously. 

3.  Joins  where  several  vectors  originating  from  one  initial  input  reunite 
and  are  combined  to  form  an  output  vector. 

4.  Interactions  where  each  element  of  a  vector  does  a  computation  based 
on  the  state  of  all  of  the  elements  of  the  vector  as  they  arrive  at  this 
point  in  the  pipe. 

5.  Outputs  where  all  of  the  elements  of  a  vector  are  gathered  together 
and  bundled  back  into  Scheme  format. 

To  use  this  kind  of  pipeline  in  solving  the  n-body  problem  it  must  be 
able  to  perform  three  operations:  scaling  a  single  vector  (scale-system), 
adding  several  vectors  (add-systems),  and  interparticle  force  calculations 
(particle-force).  Scaling  a  vector  can  be  accomplished  by  examining 
only  the  individual  elements  of  the  vector,  and  so  a  straight  pipeline  seg¬ 
ment  can  be  used.  Adding  several  vectors  is  also  done  component  by  com¬ 
ponent,  but  requires  several  different  vectors  and  hence  corresponds  to  a 
join  in  the  pipeline.  Finally,  interparticle  force  calculations  require  that 
each  component  of  a  vector  know  the  state  of  all  the  other  components  of 
the  vector  and,  thus,  is  implemented  using  a  pipeline  interaction  segment. 

The  pipeline  solution  to  the  n-body  problem  will  be  presented  in  three 
parts.  First,  in  Section  5.3.1,  the  procedures  used  to  construct  a  pipeline  are 
described  (the  details  of  the  implementation  are  presented  in  Appendix  B). 
Using  these  procedures  we  can  then  describe  the  pipeline  based  implemen¬ 
tation  of  the  procedures  scale-system,  add-systems,  particle-force, 
and  runge-kutta-4  used  in  the  original  solution  to  the  n-body  problem. 
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But  in  order  to  use  the  pipeline,  we  must  also  be  able  to  create  the  ob¬ 
jects  that  propagate  through  the  pipeline.  This  is  a  rather  complicated 
procedure,  and  is  explained  in  Section  5.3.2.  Finally,  we  can  combine  these 
operations  together  to  construct  the  remaining  major  part  of  the  n-body 
program,  integrate-system.  The  implementation  of  this  procedure  is  de¬ 
scribed  in  Section  5.3.3,  along  with  some  observations  about  its  behavior 
compared  to  that  of  the  previous  version. 

5.3.1  Building  the  Pipeline 

There  are  any  number  of  ways  in  which  a  pipeline  could  be  constructed 
to  solve  the  n-body  problem.  The  method  described  here  is  motivated  by 
two  considerations.  The  first  is  that  the  overall  modularity  of  the  original 
solution  should  not  be  changed.  That  is,  given  the  choice  of  construction 
technique,  the  code  for  runge-kutta-4  and  integrate-system  should  be 
nearly  unchanged.  The  second  consideration  is  that  the  method  should  be 
clearly  applicable  to  many  other  problems  eis  well.  In  fact,  if  the  effort 
is  not  too  great,  it  would  be  nice  to  create  a  convenient  abstraction  for 
constructing  pipelines  with  arbitrary  structure  and  operations. 

The  particular  abstraction  described  below  seems  to  meet  these  require¬ 
ments  quite  well.  Unfortunately,  the  amount  of  detail  required  to  explain 
the  implemeiiCation  is  large  enough  that  it  merely  distracts  attention  from 
the  more  important  points  that  this  example  is  intended  to  demonstrate. 
The  implementation,  therefore,  is  described  sepeirately  in  Appendix  B. 
What  follows  is  a  description  of  the  overall  nature  of  the  pipeline  eind  the 
construction  operations.  The  description  provided  here  should  be  sufficient 
for  the  purposes  of  understanding  the  remainder  of  this  example. 

There  are  two  distinct  ways  to  visualize  the  pipeline,  and  both  are  es¬ 
sential  to  understanding  its  operation.  The  first  way,  which  corresponds 
to  the  “view  from  outside”  is  that  it  has  an  input  port,  one  or  more  out¬ 
put  ports,  and  an  overall  physical  interconnection.  Specially  constructed 
vectors  of  objects  (called  pipe  vectors)  can  be  presented  at  the  input  port. 
They  progress  through  the  pipeline  and  the  objects  in  them  are  modified 
by  operational  portions  of  the  pipe  (straight  segments  and  interaction  seg¬ 
ments),  and  ultimately  arrive  at  output  ports.  The  output  ports  appear  to 
be  a  list  of  the  pipe  vectors  that  have  arrived  at  the  port  in  the  order  of 
their  arriv'al  at  this  output  port.  Naturally,  these  output  lists  will  appear 
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to  bocome  longer  and  longer  as  more  and  more  pipe  vectors  arrive  at  the 
output  port  —  in  fact,  an  attempt  to  examine  the  entire  list  will  always 
fail  to  terminate  (take  an  infinite  amount  of  time)  since  a  new  pipe  vector 
might  arrive  at  any  time. 

The  complementary  view,  “from  inside,”  is  that  a  pipe  vector  of  objects 
arrives  at  the  input  port.  Each  object  in  the  pipe  vector  progresses  inde¬ 
pendently  through  the  pipeline  —  the  pipe  vector  is  largely  a  polite  fiction 
maintained  for  the  benefit  of  outsiders.  From  the  object’s  point  of  view  the 
pipeline  is  nothing  but  a  list  of  messages  that  tell  it  how  to  mutate  its  own 
state  information  as  it  passes  through  the  pipe.  The  total  existence  of  a 
pipe  vector  consists  only  in  the  fact  that  each  object  knows  the  pipe  vector 
to  which  it  belongs  and  each  pipe  vector  knows  how  many  objects  com¬ 
pose  it  and  to  which  pipeline  input  port  it  has  been  assigned.  Section  5.3.2 
describes  the  messages  that  each  object  must  be  capable  of  handling,  and 
the  implementation  of  the  objects  themselves.  For  the  purposes  of  this  sec¬ 
tion,  we  need  only  know  that  operations  to  be  performed  on  vectors  passing 
through  the  pipe  are  actually  received  by  the  individual  objects  within  the 
pipe  vector.  Thus,  for  any  given  pipe  vector,  it  is  critical  that  all  of  the 
objects  in  that  vector  handle  the  same  set  of  messages.  It  is  these  messages 
that  the  pipeline  construction  procedures  allow  the  user  to  specify,  and  that 
consequently  tailor  the  pipeline  to  a  particular  application. 

The  work  of  constructing  the  pipeline  is  supported  by  a  package  of  pro¬ 
cedures  whose  implementation  is  discussed  in  Appendix  B.  The  following 
paragraphs  document  the  interface  to  this  package,  but  the  overall  game 
plan  must  be  explained  first.  A  pipeline  is  constructed  by  making  an  input 
port  (using  make-input).  This  is  then  extended  step  by  step:  at  each  step 
a  procedure  takes  the  previous  end  of  the  pipeline,  creates  a  new  pipe  seg¬ 
ment.  joins  them  together,  and  returns  the  newly  created  segment  (the  new 
end  of  the  pipe).  Outputs  are  created  in  a  way  similar  to  other  segments 
(by  extending  an  existing  pipe  segment  using  add-output),  although  the 
result  is  not  itself  a  pipe  segment.  You  can  imagine  the  pipeline  “growing” 
from  its  input  port  toward  its  output  ports  with  loose  ends  being  extended 
at  each  step,  leading  to  a  new  set  of  loose  ends  until  finally  these  ends  are 
“capped”  with  an  output  port. 

(make-input) 

Creates  a  pipe  input  segment.  This  is  the  way  in  which  a  pipeline 

construction  project  normally  begins.  Vectors  are  injected  into  the 
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pipeline  at  this  segment  using  start-pipeline,  discussed  in  Sec¬ 
tion  5.3.2. 

(extend  p  m) 

Given  an  existing  pipe  segment  (p)  that  is  currently  a  “dangling  end” 
of  the  pipeline,  extend  creates  a  new  straight  pipe  segment  that  will 
cause  operation  m  to  be  applied  to  vectors  passing  through  it.  This 
new  pipe  segment  is  now  a  dangling  end,  and  is  returned  to  the  caller. 
The  messages  that  can  be  used  for  m  are  defined  by  the  objects  that 
comprise  the  pipe  vectors  pcissing  through  this  segment  of  pipe. 

(fork  n  p) 

Given  an  existing  (dangling  end)  pipe  segment  (p),  return  a  list  of  n 
new  dangling  ends  for  the  pipeline.  As  a  pipe  vector  exits  segment  p 
it  is  duplicated  n  times,  and  one  copy  of  the  pipe  vector  will  continue 
along  each  of  these  output  pipes. 

(join  m  ps) 

Given  a  set  of  existing  pipe  segments  (dangling  ends,  ps),  create 
a  new  dangling  end  segment.  Each  of  the  incoming  pipe  segments 
is  extended  into  the  new  segment.  As  pipe  vectors  arrive  from  the 
input  segments  they  are  combined  using  operation  m  to  produce  a 
single  outgoing  pipe  vector  that  proceeds  down  the  newly  created 
pipe  segment.  As  in  extend,  the  operations  that  may  be  used  are 
defined  by  the  objects  which  comprise  the  pipe  vectors  arriving  at 
this  pipe  segment. 

(add-interactor  p  jn) 

Similar  to  the  operation  of  extend,  except  that  the  object  arriving  at 
this  pipe  segment  will  receive  a  record  of  the  state  of  all  the  objects  in 
the  same  pipe  vector  as  they  arrived  at  this  point.  This  permits  each 
(jbject  in  a  pipe  vector  to  compute  a  new  state  based  on  the  states  of 
all  the  other  objects  in  the  vector.  In  this  example,  add-interactor 
is  \ised  for  force  calculations. 

(add-output  p) 

Extend  an  incoming  dangling  end  by  creating  an  output  port  to  “cap” 
that  segment.  The  value  returned  by  add-output  is  a  list  of  all  of  the 
pipe  vectors  that  arrive  at  the  output  port.  In  fact,  this  value  is  at 
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all  times  a  list  whose  final  element  is  an  undetermined  placeholder.  It 
initially  contains  no  elements  at  all  (i.e.  it  is  itself  the  undetermined 
placeholder).  As  each  pipe  vector  arrives  at  the  output  port  the  un¬ 
determined  placeholder  is  determined  to  be  a  cons  of  the  new  output 
vector  and  a  new  undetermined  placeholder. 

Using  these  operations  we  can  define  our  own  specialized  pipeline  construc¬ 
tion  procedures  that  correspond  to  components  of  the  earlier  solution.  The 
first  one,  add-systems,  is  quite  simple  once  we  realize  that  it  is  used  only 
in  the  process  of  pipeline  construction.  Just  as  in  the  earlier  solution  it 
received  an  entire  system  as  input,  in  this  version  it  receives  existing  “dan¬ 
gling  ends”  of  the  pipeline.  Just  as  it  used  to  return  a  new  system,  now  it 
returns  a  new  dangling  end  of  the  pipeline: 

(define  (add-systems  .  pipes)  (join  ’add  pipes)) 

The  next  two  construction  operations  are  scale-system  and  particle- 
force.  These  were  originally  higher-order  procedures  that  produced  output 
jirocedures  which  transform  systems.  Correspondingly,  the  new  versions  are 
higher-order  procedures  that  produce  pipeline  constructors: 

(define  (scale-system  amount) 

(lambda  (pipe)  (extend  pipe  (list  ’scale  amount)))) 

(define  (particle-force  force-law) 

(lambda  (in-pipe) 

(add-interactor  in-pipe  (list  ’DERIV  force-law)))) 

Notice  that  we  have  already  begun  placing  requirements  on  the  objects 
that  will  comprise  the  vectors  passing  through  the  pii)eline  in  our  applica¬ 
tion.  We  are  requiring  that  the  objects  support  three  messages: 

1.  (scale  amount')  corresponds  to  the  procedure  scale-particle  of 
the  earlier  solution.  The  object  is  expected  to  multiply  its  state  in¬ 
formation  by  the  specified  amount. 

2.  add  corresponds  to  add-particles.  The  object  will  receive  a  list  of 
states  that  it  must  add  together  to  produce  its  new  state. 

3.  (deriv  force-law)  has  no  direct  counterpart  in  the  earlier  solution. 
The  objc-ct  will  receive  a  list  of  states  of  other  particles  in  its  vector. 
It  must  compute  the  state  of  its  differential  j^article  as  was  done  in 
the  code  for  particle-force  in  Figure  5.5  on  page  130. 
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In  addition  to  these  messages,  which  are  specific  to  the  application  of  the 
pipeline,  the  pipeline  construction  procediires  themselves  require  some  ad¬ 
ditional  messages.  These  are  described  in  Appendix  B. 

The  final  pipeline  construction  procedure  is  runge-kutta-4.  As  stated 
above,  the  overall  goal  was  that  this  procedure  should  not  change.  This 
was  not,  vmfortunately.  possible; 

(define  (runge-kutta-4  f  dt) 

...  definitions  of  dt*.  2*.  i/2*,  and  1/6*  as  in  Figure  5.6  ... 

(lambda  (y)  ;  y  is  a  PIPE 

(define  ys  (fork  5  y)) 

(define  kOs  (fork  2  (dt*  (f  (1st  ys))))) 

(define  kls  (fork  2  (dt*  (f  (add-systems 

(2nd  ys)  (1/2*  (1st  kOs))))))) 
(define  k2s  (fork  2  (dt*  (f  (add-systems 

(3rd  ys)  (1/2*  (1st  kls))))))) 
(define  k3  (dt*  (f  (add-systems  (4th  ys)  (1st  k2s))))) 
(add-systems  (5th  ys) 

(1/6*  (add-systems  (2nd  kOs)  (2*  (2nd  kls)) 

(2*  (2nd  k2s))  k3))))) 

The  changes,  however,  are  syntactic.  If  you  compare  this  procedure 
with  the  one  shown  in  Figure  5.6  on  page  132  you  will  notice  that  the  only 
differences  arise  from  the  need  to  copy  the  vector  (using  fork)  when  it 
arrives  at  the  incoming  pipe  segment.  Subsequent  references  must  specify 
an  ajrpropriate  branch.  The  procedures  1st,  2nd,  etc,  are  used  here  (as 
well  as  later)  for  the  names  of  the  appropriate  selectors  from  the  list  of 
branches.  This  requirement  arises  from  the  fact  that  a  single  variable  can 
be  referenced  in  many  places  in  a  program,  but  the  pipeline  requ’ 'es  a 
sejatrate  c(;py  for  each  reference. 

Notice  that,  just  as  the  original  version  of  runge-kutta-4  was  higher- 
order.  so  the  new  version  is  also.  That  is,  runge-kutta-4  returns  a  pro¬ 
cedure  that  will  <’xtend  a  dangling  pipe  end  (y)  into  a  newly  constructed 
])ipe  segment  and  return  the  dangling  end  of  this  new  pipeline. 

5,3.2  Pipe  Vectors  and  Objects  in  the  Pipe 

Having  labored  to  understand  the  construction  of  the  pipeline  we  can  turn 
to  the  second  task:  making  things  to  put  into  the  pipeline.  There  are. 
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in  fact,  three  separate  problems  to  address  here.  We  must  construct  the 
pipe  vectors  (make-pipe- vector),  construct  the  objects  that  comprise  the 
vectors  (make-object),  and  we  must  start  the  vectors  down  the  pipeline 
(start-pipeline).  Of  these,  the  most  complicated  one,  and  most  infor¬ 
mative,  is  the  construction  of  the  objects  themselves.  This  operation  is 
described  in  detail  in  this  section,  while  only  an  overview  of  the  others  is 
provided  here  (the  implementation  and  details  are  part  of  Appendix  B). 

As  mentioned  earlier,  vectors  in  the  pipeline  are  for  the  most  part  a 
polite  fiction  maintained  for  the  benefit  of  the  “outside  view”  of  the  pipe. 
The  real  work  is  performed  by  the  objects  that  make  up  the  vector.  A  pipe 
vector,  in  fact,  is  a  message  receiving  object  that  accepts  two  messages: 
size  returns  the  number  of  objects  in  the  vector,  and  messages  returns 
the  list  of  messages  (i.e.  the  pipeline  input  port)  to  which  that  vector  has 
been  assigned.  These  messages  are  used  only  internally  to  the  pipeline  itself, 
as  shown  in  Appendix  B.  Once  the  objects  belonging  to  a  vector  have  been 
created  using  make-object,  the  pipe  vector  itself  can  be  created  by  passing 
a  list  of  the  objects  to  make-pipe-vector.  Make-pipe-vector  does  just 
a  tiny  bit  more  than  create  the  message  receiving  object:  it  informs  each 
object  that  it  now  belongs  to  this  vector  by  a  mechanism  that  will  be 
described  along  with  the  details  of  constructing  objects. 

Once  a  pipe  vector  has  been  constructed,  it  can  be  injected  into  a  specific 
pipeline  input  port  using  start-pipeline.  As  shown  in  the  discussion  of 
Appendix  B  this  amounts  to  nothing  more  than  telling  the  vector  which 
message  list  its  objects  are  to  examine. 

With  this  background  and  overview,  it  is  time  to  discuss  the  star  players 
in  the  example:  the  objects  that  flow  through  the  pipe.  As  has  been  men¬ 
tioned  already  several  times,  the  objects  are  expected  to  accept  a  message 
from  the  pipe,  process  it,  and  continue  on  to  the  next  message.  All  objects 
must  be  able  to  handle  a  set  of  messages  that  are  used  by  the  pipeline  con¬ 
struction  operations  of  Appendix  B.  In  addition,  they  can  be  customized 
to  accept  additional  messages  appropriate  to  a  particular  application  of  the 
pipeline.  In  the  n-body  problem,  for  example,  these  messages  are  add,  de- 
riv,  and  scale  as  discussed  in  Section  5.3.1.  In  general,  an  object  consists 
of  some  internal  state  information  and  a  group  of  message  handlers  that 
mutate  this  state  in  response  to  particular  messages  received  by  the  object. 

So  far,  nothing  has  really  changed  from  the  original  solution,  except  to 
state  that  we  are  dealing  at  each  stage  with  a  vector  representing  a  system 
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of  particles  flowing  through  a  pipeline.  But  now  for  the  new  twist:  each 
object  in  a  pipe  vector  is  represented  not  by  a  static  procedure  object  (fis 
would  be  the  ordinary  practice  in  Scheme)  but  rather  by  an  active  t<isk. 
Thus,  the  pipeline  actually  operates  in  parallel  on  all  objects  within  the 
pipe  at  any  instant.  The  objects  proceed  independently  along  the  pipe 
until  they  reach  a  fork,  join,  or  output  segment.  At  a  fork,  each  object 
is  replicated  (once  for  each  output  of  the  fork)  creating  a  task  for  each 
output  direction  of  the  fork  —  thus  the  degree  of  parallelism  increases  at 
these  points.  At  a  join,  objects  wait  until  all  <  f  their  partners  arrive  on  the 
other  input  paths  and  then  a  single  object  is  created  to  travel  along  the 
output  path.  At  an  output,  all  of  the  objects  from  a  single  input  vector 
are  gathered  together  and  then  are  released  as  a  unit.  The  pipeline  system 
is,  in  fact,  built  out  of  active  objects  flowing  through  a  statically  created 
pipeline. 

This  new  approach  has  several  advantages  over  the  origin^ll.  First,  by 
creating  processes  only  when  an  object  is  created  or  at  a  fork  in  the  pipeline 
we  can  amortize  the  cost  of  process  creation  over  the  straight  line  length 
of  a  pipe.  Thus  operations  that  would  not  merit  the  creation  of  a  task 
for  each  particle  in  the  original  implementation  can  be  executed  in  parallel 
without  remodularizing  the  program  to  combine  them  into  larger  (hence 
longer  duration)  operations. 

Furthermore,  since  each  object  is  an  independent  task  traveling  along 
the  pipeline,  the  system  need  not  idle  while  one  object  is  undergoing  lengthy 
processing.  The  objects  do  not  travel  in  lock-step  through  the  system  as 
they  w’ould  in  a  traditional  vector  processor,  and  it  would  be  simple,  in  fact, 
to  allow  the  pipeline  to  contain  conditional  branches  or  loops  that  would 
affect  only  some  of  the  objects  in  any  given  input  vector. 

To  make  the  job  of  creating  active  objects  for  inclusion  in  pipeline  vec¬ 
tors  simpler,  make-object  creates  the  appropriate  task  and  handles  the 
standard  messages  required  of  every  object  (see  Appendix  B  for  a  descrip¬ 
tion  of  the.se  messages).  In  order  to  make  an  object  that  has  its  own  state 
and  specialized  messages  we  merely  specify  the  initial  state  of  the  object,  a 
procedure  that  will  handle  object-specific  messages,  and  a  procedure  that 
will  copy  the  state  of  the  object.  As  explained  in  Section  5.3.1  the  objects 
in  our  example  must  handle  three  kinds  of  messages. 

Beff)re  df'seribing  the  code  for  make-object  (shown  in  Figure  5.11)  it 
will  help  to  see  how  it  is  used  in  this  particular  example.  This  will  sharpen 
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our  understanding  of  its  specification,  and  thereby  make  the  actual  imple¬ 
mentation  considerably  easier  to  absorb.  Because  the  pipe  vector  (which 
propagates  through  the  pipeline)  corresponds  to  the  systems  of  the  earlier 
solution,  we  will  require  two  kinds  of  objects.  These  correspond  to  the  ele¬ 
ments  of  a  system:  the  time  (whose  state  is  a  simple  number),  and  particles 
(with  a  more  complex  state  including  name,  mass,  position,  and  velocity). 
Time  objects  are  simple  to  create,  given  make-object: 

(define  (make-time-element  time) 

(define  (time-messages  time  m  arg) 

(cond  ((eq?  m  ’ADD)  (apply  +  arg)) 

((eq?  (1st  m)  ’DERIV)  1)  ;  DERIV:  dt/dt 
((eq?  (1st  m)  ’SCALE)  (*  time  (2nd  m))))) 

(make-object  time  ;  Initial  State 

(lambda  (x)  x)  ;  Copy  State 

time-messages))  ;  Message  Handler 

Notice  the  way  the  message  handler  is  expected  to  operate.  It  will 
receive  three  arguments:  the  current  state  of  the  object  (time  in  this  ex¬ 
ample),  the  message  to  be  processed  (m),  and  a  single  additional  argument 
(arg).  It  is  expected  to  update  the  state  information  and  return  the  new 
state.  It  may  do  this  by  side-effecting  a  data  structure  or  by  creating  a  new 
one  depending  upon  implementation  considerations. 

The  additional  argument,  arg,  has  three  different  meanings  depending 
upon  which  operation  was  used  to  create  the  pipe  segment  sending  the 


Extend 

The  argument  is  always  ’  ().  Any  information  needed  to  process  the 
message  must  be  included  in  the  message  itself  and  the  state  of  the 
object.  For  this  reason  scale-system  (on  page  145)  includes  the 
scale  factor  as  part  of  the  message. 

Join 

The  argument  is  a  list  of  the  states  of  corresponding  objects  arriving 
at  the  junction. 

Interaction 

The  argument  is  a  list  of  the  states  of  all  the  objects  in  the  same 
v(’rtor  as  this  object. 
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For  particles,  the  state  is  the  same  as  the  data  structure  used  in  the  se¬ 
rial  implementation  of  particles  (a  standard  Scheme  vector).  The  defini¬ 
tion  of  scale-particle !  and  add-particles!  (used  in  the  code  below)are 
omitted;  they  are  simply  side-effecting  versions  of  the  procedures  scale- 
particle  and  add-particles  shown  in  Figure  5.3®. 

(define  (make-particle-element  particle) 

(define  (interact  force-law  me  all) 

(make-particle  (name  me)  0  ;  name  and  dm/dt 

(velocity  me)  ;  dx/dt 

(reduce  add  ;  dv/dt 

(map  (lambda  (other)  (force-law  me  other)) 

(cdr  all)))))  ;  Ignore  time! 

(define  (particle-messages  state  m  arg) 

(cond  ((eq?  m  ’ADD)  (add-particles!  state  arg)) 

((eq?  (1st  m)  ’SCALE)  (scale-particle!  state  (2nd  m))) 
((eq?  (1st  m)  ’DERIV)  (interact  (2nd  m)  state  arg)))) 

(make-object  particle  ;  Initial  State 

vector-copy  ;  Copy  State 

particle-messages))  ;  Message  Handler 

The  core  of  the  procedure  Make-Object  is  shown  in  Figure  5.11.  The 
code  to  handle  most  messages  has  been  omitted  since  it  is  quite  straight¬ 
forward.  Only  three  messages  are  handled  by  the  code  shown  in  the  figure: 

•  Procedural  messages  are  used  in  the  implementation  of  the  pipeline, 
as  shown  in  Appendix  B.  An  object  (i.e.  task)  responds  to  a  pro¬ 
cedural  message  by  calling  the  procedure  and  supplying  the  object’s 
own  message  handler.  This  allows  the  procedure  to  effectively  send 
multiple  messages  to  the  object  without  requiring  the  creation  of  a 
complicated  and  lengthy  pipe  segment. 

•  Halt  indicates  that  the  object  (task)  should  terminate  immediately. 
It  is  used  both  for  the  output  port  of  a  pipeline  and  for  pipeline  joins. 
See  Appendix  B  for  examples. 

'’The  u.se  of  (cdr  all)  to  ignore  the  time  of  the  system  when  doing  the  interparti- 
cle  force  calculation  corresponds  to  the  construction  of  the  pipe  vector  in  the  procedure 
serial->pipe-system,  shown  on  page  154. 
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•  User  messages  are  those  not  required  of  all  objects.  They  are  han¬ 
dled  by  calling  the  user  supplied  message  handler  (for  example,  the 
procedure  time-messages  for  time  objects  created  by  make-time- 
element).  They  pass  along  the  current  state  of  the  object,  the  mes¬ 
sage,  and  any  additional  argument  that  may  have  been  specified.  The 
user's  message  handler  is  expected  to  return  the  new  state  of  the  ob¬ 
ject. 

One  final  note  before  diving  into  the  implementation  details.  In  addition 
to  the  job  of  creating  a  task  for  the  object,  make-object  returns  a  Scheme 
procedure  that  can  be  used  to  activate  the  task.  The  task  is  created  in 
such  a  way  that  it  will  touch  an  undetermined  future  almost  immediately 
after  creation,  and  hence  will  become  inactive.  The  procedure  returned  by 
make-object  allows  the  task  to  become  active  once  the  object  is  made  part 
of  a  pipe  vector  and  the  pipe  vector  is  placed  in  the  pipeline. 

The  following  items  are  keyed  to  the  implementation  of  make-object 
shown  in  Figure  5.11. 

1.  The  procedure  make-placeholder  creates  a  placeholder  with  no  as¬ 
sociated  task.  As  a  result,  any  task  that  touches  it  will  hang  until 
some  other  task  explicitly  gives  it  a  value  (using  determine!).  It 
is  used  here  as  the  initial  list  of  messages  to  be  processed  by  this 
object  so  that  the  task  associated  with  this  object  will  wait  until  it 
is  inserted  into  a  pipeline.  This  conveniently  implements  a  “start 
gate”  to  hold  off  execution  until  a  selected  time.  It  receives  a  value 
somewhat  indirectly,  but  the  start  of  that  process  is  in  step  7. 

2.  This  clause  handles  procedural  messages  by  providing  them  with  the 
message  handler  itself.  The  task  that  implements  the  object  exe¬ 
cutes  the  procedural  message  and  thus  sends  itself  messages  before 
proceeding  on  to  the  next  message  in  the  pipeline. 


3.  (See  also  step  5.)  The  procedure  kill-task  is  not  shown.  It  removes 
the  task  that  implements  the  object  from  the  list  of  active  objects 
in  the  system  and  then  releases  the  processor  to  other  tasks,  “com¬ 


mitting  suicide.”  This  list  of  active  objects  is  essential  to  the  correct 
operation  of  the  pipeline.  As  described  in  Section  3.4,  a  task  contih- 
ucs  to  compute  only  as  long  as  the  placeholder  which  is  its  goal  is 
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(define  (make-object  state  copy-state  user-code) 

(let  ((my-cell) 

(vector) 

(ob  j  ect-n\imber ) 

(the-messages  (make-placeholder)))  ;  (l) 

(define  (loop  message-list) 

(define  (standard-handler  m  #! optional  arg) 

(cond 

((procedure?  m)  (m  standard-handler))  ;  (2) 

((eq?  m  ’HALT)  (kill-task  my-cell))  ;  (3) 

. . .  more  standard  message  handlers  . . . 

(else  ;  (4) 

(set!  state 

(user-code  state  m  (if  (unassigned?  arg)  ’()  arg)))))) 
(standard-handler  (car  message-list)) 

(loop  (cdr  message-list))) 

(set!  my-cell  ;  (5) 

(make-task  (lambda  ()  (loop  the-messages)))) 

(lambda  (the-vector  niimber  messages)  ;  (6) 

(set!  vector  the-vector) 

(set!  object-number  number) 

(determine!  the-messages  messages))))  ;  (7) 

See  text  beginning  on  page  151  for  footnotes. 

Figure  5.11:  Creating  Vector  Elements 
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refon'iicod  by  the  system.  But  the  pipeline  works  by  using  tasks  that 
have  no  goal  plaeeliolder.  Thus  the  pipeline  operations  themselves 
maintain  a  globally  visible  data  structure  (a  doubly  linked  list)  of  ob¬ 
jects  still  in  the  pipe  in  order  to  guarantee  that  they  will  remain  in  the 
system  after  garbage  collection.  The  variable  my-cell  is  initialized 
when  the  task  is  created. 

4.  Messages  other  than  the  standard  ones  are  handled  by  invoking  the 
user's  procedure  user-code. 

5.  (See  also  step  3.)  The  procedure  madce-task  is  not  shown.  It  creates 
a  task  with  no  goal  placeholder  and  places  it  on  a  globally  accessible 
data  structure  so  that  the  ttisk  will  not  be  garbage  collected.  A 
pointer  to  the  appropriate  part  of  this  global  data  structure  is  stored 
in  the  variable  my-cell  for  use  when  the  object  receives  the  halt 
message.  This  task  begins  immediately  by  calling  loop  to  process 
messages  from  the-messages.  Recall  from  step  1  that  this  is  initially 
a  placeholder  with  no  value,  so  that  the  task  runs  for  a  very  short  time 
before  becoming  inactive  while  waiting  for  the  messages  to  arrive. 

G.  This  is  the  procedure  actually  returned  from  the  call  to  make-object. 
It  is  u.sed  only  by  make-pipe-vector  (see  Appendix  B),  and  specifies 
the  vector  to  which  this  object  belongs,  the  offset  within  that  vector, 
and  the  messages  which  that  vector  is  to  process. 

7.  The  task  associated  with  this  object  will  be  able  to  proceed  after  the 
call  to  make-pipe-vector  since  this  determine!  operation  supplies 
a  value  for  the  placeholder  described  in  steps  1  and  5. 

.\t  thi.s  point  we  have  implemented  almost  all  of  the  pieces  we  need  to  make 
the  example  work.  We  can  create  objects  representing  particles  (make- 
particle-element)  and  times  (make-time-element),  combine  them  into 
pipe  vectors  representing  systems  (make-pipe-vector),  and  start  them 
througli  the  pipeline  (start-pipeline).  The  only  remaining  jejb  is  to 
put  these  pieces  together  to  provide  the  simplified  interface,  integrate- 
system. 
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5.3.3  Integration,  Pipeline  Style 

The  procedure  integrate-system  is  responsible  for  initializing  the  pipeline 
and  keeping  it  busy.  Notice  that  integrate-system  is  defined  (in  Sec¬ 
tion  5.2)  to  return  a  standard  stream  of  results,  beginning  with  the  initial 
state  of  the  .system.  The  solution  presented  here  retains  this  interface,  al¬ 
though  doing  so  is  slightly  more  difficult  than  in  the  earlier  implementation. 

Before  proceeding  to  describe  the  new  implementation  of  integrate- 
system  there  are  two  small  jobs  to  perform.  First,  review  the  implemen¬ 
tation  of  integrate-system  shown  in  Figure  5.6  on  page  132.  The  new 
version  will  be  similar,  but  differs  in  some  significant  ways.  Second,  let’s 
review  the  pieces  we  have  in  our  newly  constructed  pipeline  arsenal. 

From  Section  5.3.1  (and  .Appendix  B)  we  have  the  standard  pipeline 
constructor  procedures  make-input  (for  creating  an  input  port)  and  add- 
output  (for  extending  a  pipe  into  an  output  port  that  appears  as  an  ever- 
growing  list).  Using  these,  we  implemented  (in  Section  5.3.1)  some  proce¬ 
dures  that  construct  pipeline  segments  specific  to  our  needs.  Particle- 
force  takes  a  force  law  (in  our  cause  we  will  use  the  same  gravitation 
law  used  in  the  .solution  of  Section  5.2)  and  produces  a  procedure  that  will 
extend  an  existing  pipeline  by  fitting  on  a  new  segment  that  performs  the 
appropriate  interaction  calculation.  Similarly,  Runge-kutta-4  takes  such  a 
pipeline  extender  procedure  and  a  time  step  and  returns  a  procedure  that 
will  extend  an  existing  pipeline  into  an  integrator  pipeline. 

We  also  have  some  procedures  from  Section  5.3.2.  We  will  use  these  to 
create  two  new  utility  routines.  The  first  takes  a  system  of  particles  in  the 
form  used  in  Section  5.2  and  constructs  a  pipe  vector  from  them: 

(define  (serial->pipe-system  serial-system) 

(make-pipe- vector 

(cons  (make-time-element  (time  serial-system)) 

(map  make-particle-eleraent  (particles  serial-system))))) 

This  is  simple  enough.  It  converts  the  time  and  particles  from  the  se¬ 
rial  system  into  the  corresponding  time  and  particle  objects  for  the  pipeline 
(  using  make-time-element  and  make-particle-element  of  Section  5.3.2). 
These  are  then  combined  together  into  a  single  pipe  vector  using  make- 
pipe-vector.  Notice  that  it  is  at  the  time  when  this  procedure  is  called 
that  most  of  the  activt'  objects  (ta.sks)  are  created  (the  remainder  are  ere- 
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ated  when  these  objects  arrive  at  the  forks  in  the  pipeline  created  by  runge- 
kutta-4). 

The  inverse  operation  is  even  simpler.  The  entry  that  appears  in  the 
list  at  the  output  port  of  a  pipeline  when  a  pipe  vector  arrives  at  the  port 
is  just  a  list  of  the  state  information  of  the  objects  that  were  part  of  the 
pipe  vector.  Since  at  the  input  we  made  the  time  object  be  the  first  entry 
in  the  pipe  vector,  it  will  be  the  first  entry  in  the  output  as  well: 

(define  (pipe->serial-system  pipe-system) 

(make-system  (car  pipe-system)  (cdr  pipe-system))) 

The  only  other  operation  we  need  was  described  in  Section  5.3.2  and 
shown  in  Appendix  B.  This  is  start-pipeline  which  requires  a  pipeline 
input  port  and  a  pipe  vector,  and  merely  inserts  the  vector  into  the  pipeline. 

With  these  two  jobs  completed  we  can  discuss  the  implementation  of 
integrate-system  shown  in  Figure  5.12.  Notice  that  the  outline  of  the 
pipeline  version  of  integrate-system  and  the  version  in  Figure  5.6  are 
roughly  the  same.  The  new  version  is  more  complicated,  but  it  still  consists 
of  two  parts.  The  first  constructs  the  pipeline  and  starts  it  operating.  The 
second  is  a  procedure  to  compute  subsequent  values  of  the  system  state. 

The  differences  between  the  two  versions  arise  from  two  sources.  In 
the  pipeline  version,  it  is  necessary  to  construct  the  actual  pipeline  when 
integrate-system  is  called,  since  it  is  only  then  that  all  of  the  compo¬ 
nents  are  available.  Furthermore,  two  different  peu-ts  of  the  pipeline  must 
be  located  for  future  use  (its  input  and  output  ports).  These  make  the 
construction  somewhat  more  cumbersome  than  in  the  earlier  version.  It  is 
further  complicated  by  a  technical  point:  it  would  be  undesirable  to  allow 
the  original  output  port  of  the  pipeline  to  be  visible  once  the  value  of  the 
system  after  the  first  time  step  has  been  extracted.  If  this  were  done,  then 
the  garbage  collector  would  “see”  the  potentially  infinite  list  of  states  of 
the  system  over  time  —  instead  it  is  maintained  in  a  local  variable  that  be¬ 
comes  inaccessible  after  the  first  call  to  next.  This  is  a  standard  precaution 
used  in  Scheme  when  dealing  with  potentially  infinite  streams. 

The  second  difference  is  that  the  pipeline  must  be  started  and  kept  busy. 
This  is  done  with  the  two  calls  to  start-pipeline.  When  integrate- 
system  is  called,  the  initial  state  of  the  system  is  fed  into  the  input  port 
of  the  pipeline.  Every  time  a  result  is  extracted  from  the  system  (by  a  call 
fo  next)  the  result  is  also  fed  back  into  the  pipe  so  that  the  pipeline  can 
begin  computing  the  system  state  at  the  next  time  step  if  it  is  called  for. 
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(define  (integrate-system  initial-state  dt) 

(define  integrator-maker  ;  (1) 

(runge-kutta-4  (particle-force  gravitation)  dt)) 

(define  input  (make-input)) 

(define  (next  output-list) 

(let  ((next-out  (pipe->serial-system  (car  output-list)))) 
(start-pipeline  ;  (2) 

input  (serial->pipe-system  next-out)) 

(cons  next-out  ;  (3) 

(delay  (next  (cdr  output-list)))))) 

(let  ((pipeline  ;  (4) 

(add-output  (integrator-maker  input)))) 

(start-pipeline  ;  (5) 

input  (serial->pipe-system  initial-state)) 

(cons  initial-state  ;  (6) 

(delay  (next  pipeline))))) 

Notes: 

1.  Integrator-Maker  will  take  an  input  pipeline  “dangling  end”  and  extend  it  into  a 
newly  constructed  integration  pipeline. 

2.  As  each  result  is  pulled  from  the  stream  returned  by  integrate-system  the  output 
system  is  turned  around  and  started  back  down  the  pipeline.  This  allows  the 
pipeline  to  overlap  computation  of  the  next  answer  with  the  handling  of  the  current 
answer.  See  also  comment  5. 

3.  Create  the  actual  item  to  be  added  to  the  stream  of  answers.  As  in  most  stream 
processes,  this  is  a  cons  of  the  currently  available  answer  and  a  promise  to  compute 
the  next  one. 

4.  Create  the  complete  pipeline  by  taking  the  input  port  (input),  extending  it  with 
an  integration  pipeline,  and  capping  that  with  an  output  port, 

0.  Start  the  pipeline  computing  the  state  of  the  system  one  time  step  into  the  future. 
See  also  comment  2. 

The  result  returned  by  integrate-system  is  the  current  state  and  a  promise  to 
compute  the  next  state. 


Figure  5.12:  Pipeline  Version  of  Integrate-System 
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Tills  implementation,  in  some  sense,  has  a  bit  of  speculative  computa¬ 
tion.  It  computes  one  time  step  ahead  of  what  has  been  requested.  Notice 
that  this  is  not  true  of  the  earlier  solution,  and  as  a  result  performance 
of  the  two  systems  is  somewhat  hcird  to  compare.  To  further  complicate 
the  measurement  problem,  notice  that  integrate-system  in  the  pipeline 
approach  is  responsible  for  constructing  the  pipeline.  Since  the  pipeline  is  a 
static  structure  it  seems  hardly  fair  to  count  this  cost  as  part  of  the  compu¬ 
tation  (since  it  is  amortized  over  the  total  number  of  time  steps  ultimately 
performed ) . 

If  the  intention  of  the  program  is  to  continue  the  calculations  indefi¬ 
nitely  into  the  future  (rather  than  explicitly  by  demand),  we  could  have 
constructed  a  looping  pipeline  by  placing  a  fork  in  the  pipeline  after  the 
integrator  segment.  We  would  then  feed  one  end  of  this  fork  into  the  out¬ 
put  port  and  “weld”  the  other  back  to  the  pipe  segment  leading  into  the 
integrator.  This  would  lead  to  unbounded  speculative  computation,  since 
the  objects  in  the  pipeline  would  then  be  able  to  loop  forever.  The  im¬ 
plementation  shown  here  would  need  some  modifications  to  support  this 
possibility. 

5.3.4  Parallel  Pipeline:  Performance 

Unlike  the  n-body  programs  in  Section  5.2,  this  pipeline  implementation 
ha.s  performance  characteristics  that  are  difficult  to  compare  directly  with 
the  serial  version.  In  attempting  to  make  this  comparison,  a  number  of 
issues  ari.se  that  are  interesting  in  their  own  right.  The  results  reported  in 
this  section  reflect  only  a  cursory  attempt  to  understand  the  performance 
of  this  pipeline  emulation  —  they  are  intended  (in  the  same  vein  as  the 
examples  themselves)  to  illustrate  how  such  measurements  can  be  made. 
This  highlights  the  more  important  point  that  MultiScheme  provides  a  good 
liase  for  emulating  other  systems,  a.s  discussed  in  Section  5.4. 

First,  the  good  news.  The  program  presented  here  was  able  to  solve  the 
problem  a  factor  of  6.9  times  faster  than  the  original  serial  version  —  run¬ 
ning  somewhat  faster  than  the  final  version  of  the  program  of  Section  5.2, 
although  slower  than  the  modified  version  that  motivated  this  approach^ 


'  Hi'cause  of  niacliine  availability,  tlie  ineaisureiTients  in  Section  5.2  were  taken  separately 
from  tlio.se  in  Section  5.3.  The  systems  used  to  collect  the  measurements  differed  in  the 
total  number  of  physical  procc.ssors  and  amounts  of  memory,  although  the  experiments 
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By  adding  measurement  hooks  to  the  pipeline  implementation  of  Ap¬ 
pendix  B,  it  is  possible  to  gain  more  detailed  insight  into  this  approach. 
For  example,  running  one  sample  data  set  through  the  pipeline  to  perform 
a  single  integration  step  yields  the  following  breakdown  of  processor  time®; 


Time 
(kticks  ) 


Description 


Pipeline  Message  Handling 

424.1  Handling  the  pipeline  join  message. 

12.0  Handling  the  pipeline  fork  message. 

112.6  Handling  the  pipeline  interact  message. 

0.3  Handling  the  pipeline  output  message. 

User  Message  Handling 

(includes  time  spent  waiting  for  value  of  placeholders) 

535.2  Calling  user  supplied  message  handlers  from  straight 
pipe  segments. 

419.1  Calling  user  supplied  handler  as  part  of  the  join  op¬ 
eration. 

110.6  Calling  user  supplied  handler  ais  part  of  the  interact 
operation. 

Pipeline  Internal  Utilities 

5.8  Searching  and  inserting  into  tables,  part  of  the  join, 
output,  and  interact  operations. 

7.3  Busy-waiting  on  locks,  part  of  the  table  operations 
and  process  creation  and  removal. 


were  performed  iKsing  the  same  numbers  of  active  processors.  The  speed-up  meaisurements 
are  ratios  of  measurements,  both  of  which  are  taken  from  the  same  system.  These  ratios, 
therefore,  should  be  comparable  even  though  the  raw  data  from  which  they  are  calculated 
is  not. 

"The  timing  units  here  are  “kiloticks,”  1000  times  the  amount  of  time  required  to  call 
and  return  from  the  procedure  x  defined  by  (define  (x)  3).  This  time  unit  is  relative 
insensitive  to  the  actual  machine  on  which  the  measurements  are  taken.  For  comparison, 
see  Appendix  C. 
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In  addition,  the  real  time  required  to  compute  the  new  system  state 
was  16.5  kiloticks.  Since  the  computation  was  done  using  16  processors, 
this  indicates  that  a  total  of  264  kiloticks  of  CPU  time  were  available. 
The  measured  idle  time,  totalled  across  all  processors,  was  154.1  kiloticks. 
Thus  the  actual  amount  of  time  spent  computing  was  264.0  —  154.1  or  109.9 
kiloticks. 

We  can  condense  this  information  to  gain  an  estimate  of  the  cost  of 
implementing  the  actual  pipeline  operations.  For  example,  to  compute  the 
percentage  of  time  spent  in  the  overhead  of  a  join  operation,  we  take  the 
424.1  kiloticks  required  for  all  joins  and  subtract  the  amount  of  time  spent 
in  the  user’s  code  that  actually  combines  the  incoming  data,  a  total  of  419.1 
ticks.  The  pipeline  overhead  is  thus  50  kiloticks,  or  a  total  of  4.5%  of  the 
CPU  time.  Performing  this  calculation  for  the  other  operations  yields  the 
following  picture  of  processor  utilization: 


%  Time 

Description 

10.9 

Pipeline  fork  operations 

4.5 

Pipeline  join  operations 

1.8 

Pipeline  interact  operations 

0.2 

Pipeline  output  operations 

82.6 

User  Program 

The  S2.6%  of  processor  time  available  to  the  user  program  can  be  further 
subdivided.  While  actual  measurement  of  the  amount  of  time  spent  in  user 
code  (as  opposed  to,  for  example,  the  MuItiScheme  scheduler  and  the  cost 
of  message  dispatching)  is  a  difficult  t^lsk,  we  can  still  make  a  reasonable 
estimate.  Since  the  serial  version  took  732.1  kiloticks  to  execute  (if  we 
disregard  garbage  collection  which  did  not  occur  in  the  parallel  case),  we 
can  assume  that  roughly  the  same  amount  of  time  was  required  by  the 
pipeline  execution  of  the  program.  This  732.1  kiloticks  comes  to  66.6%  of 
the  total  CPU  time  spent  in  the  program,  leaving  16.0%  for  unallocated 
overhead. 

Finally,  as  can  be  seen  from  this  table,  17.4%  of  the  total  time  was 
spent  in  the  pipeline  operations  themselves.  Yet  from  the  previous  table 
(under  the  heading  “pipeline  internal  utilities)  only  5.3%  of  the  total  time 
is  spent  manipulating  the  tables  used  for  matching  objects  arriving  from 
different  branches  of  the  pipe  (at  joins  and  interaction  points).  At  the  same 
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time,  0.0%  of  the  total  time  goes  into  serializing  access  to  data  structures, 
indicating  that  the  creation  and  destruction  of  tasks  represents  a  more 
serious  problem  than  the  matching  itself. 

5.3.5  Parallel  Pipeline;  Summary 

This  final  example  has  served  to  demonstrate  three  points.  It  clearly  shows 
the  independent  utility  of  tasks,  placeholders,  and  the  determine!  oper¬ 
ation.  But  notice  that  tasks  (without  goal  placeholders)  are  used  to  im- 
lilement  the  objects  that  propagate  through  the  pipeline  (step  5  of  make- 
object.  on  page  153).  On  the  other  hand,  placeholders  (witliout  tasks)  are 
used  to  function  as  "gates"  that  allow  tasks  to  be  conveniently  activated 
using  the  determine!  operation  (steps  1  and  7  of  make-object). 

Second,  while  the  details  have  not  been  shown,  notice  that  the  decision 
to  u.se  tasks  without  specific  computation  goals  makes  the  garbage  collector 
a  danger  to  the  computation.  Special  arrangements  (hidden  inside  make- 
task  and  kill-task  in  steps  3  and  5  of  make-object)  are  required  to 
maintain  a  global  data  structure  that  specifies  the  tasks  that  are  in  the 
pipeline  lest  they  be  removed  at  the  next  garbage  collection.  Unfortunately, 
thi.s  data  structure  can  retain  tasks  that  are  no  longer  useful.  For  example, 
if  the  user  stops  all  tasks  using  pause-everything  and  then  discards  the 
tasks  which  were  running  (an  operation  that  is  particularly  convenient  since 
if  coi  res{)onds  to  the  standard  ‘'abort"  interrupt  handler)  the  data  structure 
will  still  contain  rt'fereuces  to  the  tasks.  These  tasks  will  not  drain  processor 
resovirces  (they  are  no  longer  candidates  for  processor  time)  although  the 
data  structures  that  suppent  them  will  remain  in  memory. 

But  most  importantly  this  example  shows  that  MultiScheme’s  packaging 
of  i)arallel  computation  allows  a  radically  different  approach  to  parallel  pro¬ 
gramming  to  ije  emi)edded  within  the  language.  To  emphasize  this  point,  it 
i,-.  worth  reviewing  this  solution  to  the  n-body  problem  from  this  partievdar 
ju)inf  of  \  iew.  B(';ir  in  mind  that  the  pipeline  jjrocedures  of  Appendix  B 
together  with  make-object  of  Section  5.3.2  represent  the  interface  to  the 
embed<led  computation  model. 

ITing  these,  we  easily  created  four  procedures  that  construct  a  pipeline 
'ui'able  for  the  i)arti<ad:u'  ijroblem  under  consideration  (Section  5.3.1).  In 
onli  i  to  ttiilor  the  opertitions  of  the  pipeline  to  our  own  application,  we 
added  two  more  procedures  that  construct  objects  with  properties  specific 
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t(i  our  application  (Section  5.3.2).  We  created  two  procedures  to  convert 
l)('tu  e('n  tlie  serial  represeuitations  of  particle  systems  that  the  user  normally 
deals  with  and  the  pipe  vectors  which  propagate  through  the  pipeline  (Sec¬ 
tion  •j.3.3).  Finally,  w(’  wrote  the  only  complicated  procedure  (integrate- 
system  i  which  serves  to  keep  the  pipeline  full  and  convert  its  results  into 
a  uu'ie  ^randard  r<‘]uesentation'^.  In  fact,  the  total  amount  of  code  needed 
to  solve  t'le  prohhnn  using  this  interface  amounts  to  less  than  two  pages  of 
text,  and  mu.  h  of  this  is  nearly  unchrmged  from  the  original  implementa- 

tli  Hi. 

This  work,  once  tlu'  pipeline  interface  is  specified,  is  not  difficult.  In 
fact,  it  seems  likely  that  anj'one  with  a  reasonable  familiarity  with  Scheme 
and  the  application  itself  (i.e.  the  use  of  a  Runge-Kutta  integrator  to  solve 
the  n  Ijody  problem)  could  be  expected  to  construct  the  solution.  The 
embc'dding  of  tlie  pipeline  is  a  more  significant  project  as  is  only  to  be 
expected.  Yet,  as  can  be  seen  from  the  code  in  Appendix  B,  even  this 
prt^ject  is  not  hard  once  the  interface  is  defined. 


5.4  Other  Methodologies 

This  section  shows  how  three  additional  parallel- programming  methodolo¬ 
gies  can  be  expressed  in  MultiScheme.  Section  5.4.1  briefly  reviews  the 
extensions  to  Lisp  proposed  by  Gabriel  and  McCarthy[23]  for  introduc¬ 
ing  parallelism,  and  then  presents  a  set  of  syntactic  rewrite  rules  (macros) 
rhat  provide  these  same  constructs  within  MultiScheme.  Section  5.4.2  out¬ 
lines  a  common  approach  for  command-based  (as  opposed  to  expression- 
based)  languages,  the  fork  and  join  construct.  Again,  this  is  easily  embed¬ 
ded  within  MultiScheme,  although  the  “linguistic  fit”  of  these  constructs 
is  rather  poor  in  an  expression-based  language  like  Lisp.  Finally,  Sec¬ 
tion  5.4.3  discusses  the  approach  to  parallelism  advocated  by  BBN  for  the 
C  languagefoG]. 


'In  fact,  this  final  proccflure  is  probably  a  sufficiently  common  type  of  operation  that 
it  would  be  provided  as  a  part  of  the  interface  to  the  pipeline  package  in  a  more  developed 
implementation. 
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5.4.1  QLambda 

In  the  suininer  of  19S4,  prior  to  the  work  on  MultiSchenie,  Gabriel  and 
McCarthy  suggested  a  trio  of  extensions  to  Lisp[23]  intended  to  su])p()rt 
parallel  prograinnhng.  In  their  language,  the  special  forms  lambda  and 
let  have  matching  counterparts,  qlaimbda  and  qlet,  that  allow  the  user 
to  sjiecify  how  eagerly  the  body  of  the  expression  is  to  be  evaluated.  The 
ba-^ic  model  is  one  of  indefiendent  tasks  communicating  by  a  queue  of  inj)nt 
argtime’iits.  Evaluation  of  a  qlambda  expression  leads  to  a  new  task  running 
indepeiuh'iitly.  A  call  to  this  task  is  performed  by  evaluating  the  arguments 
to  the  task  and  [tutting  them  on  the  task’s  queue.  Evaluation  of  a  qlet 
allows  the  values  of  the  bound  variables  to  be  calculated  in  parallel  with 
the  execution  of  the  body  of  the  qlet  expression. 

Each  of  these  constructs  can  operate  at  three  eagerness  levels  (my  names 
for  the  levels,  not  theirs);  none,  providing  Itehavior  identical  to  the  standard 
(se(iuential)  special  forms;  normal,  creating  a  task  that  waits  for  all  of  tlu' 
argument  values  to  arrive  before  proceeding  on  to  execute  the  body  in 
parallel  with  the  call  to  it;  or  eager,  creating  a  task  and  executing  the 
body  in  parallel  with  the  task  pausing  only  when  it  attempts  to  reference 
arguments  that  have  not  yet  arrived.  Gabriel  and  McCarthy  propos('d  a 
similar,  but  more  complicated,  extension  for  catch  and  throw. 

In  March  of  1985.  with  the  help  of  Byron  Davies  (a  member  of  the  group 
at  Stanford  that  w’as  developing  QLisp),  a  set  of  macros  w'as  develojied 
for  converting  programs  written  using  qlet  and  qlambda  into  standard 
MultiScheme  programs.  No  support  was  provided  for  qcatch,  since  its 
definition  was  still  vmder  active  development  at  the  time. 

The  results  of  this  work,  somewhat  revised,  are  described  below'.  Each 
of  the  six  cases  (two  s[)ecial  forms,  each  with  three  eagerness  values)  is 
described.  In  QLis[),  th<'  amount  of  eagerness  may  be  specified  either  sym 
tactically  (by  using  a  [uirticular  constant  for  each  of  the  three  values)  or  at 
runtime  (by  using  a  n(Ui-constant  expression).  The  rewrite  rules  exi)and  as 
show'll  if  the  amount  of  eagerness  is  constant.  Otherwise,  they  exjiaiul  into 
a  runtiiiK'  case  disjiatch  to  the  correctly  exiianded  version. 

Converting  qlet  Expressions 
qlet  ex[)ression  has  th<‘  form 
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(qlet  <e>  (<bindings>)  .  <body>) 

where  <e>  specifies  the  amount  of  eagerness  and  each  <binding>  consists 
of  a  variable  and  an  expression.  To  make  the  explanation  simpler,  we  will 
expand  a  particular  qlet  expression; 

(qlet  e  ((a  (f  3)) 

(b  (g  4))) 

(cons  a  b)) 

If  the  eagerness  is  none,  this  converts  to  the  standard  let  special  form: 

(let  ((a  (f  3)) 

(b  (g  4))) 

(cons  a  b)) 

If  the  eagerness  is  normal,  then  the  intent  is  to  evaluate  the  expressions 
in  the  <bindings>  in  parallel.  When  all  of  these  vadues  axe  available,  the 
body  can  then  be  evaluated: 

(let  ((a  (future  (f  3))) 

(b  (future  (g  4)))) 

(touch  a) 

(touch  b) 

(cons  a  b)) 

Finally,  an  eagerness  of  eager  indicates  that  execution  of  the  body  should 
proceed  concurrently  with  evaluation  of  the  expressions  in  the  <bindings>: 

(let  ((a  (future  (f  3))) 

(b  (future  (g  4)))) 

(cons  a  b)) 

Notice  that  in  this  eager  ca.se  the  value  (a  cons  cell)  may  well  be  returned 
to  the  caller  long  before  the  values  of  (f  3)  and  (g  4)  are  computed.  Since 
cons  does  not  touch  its  arguments  the  cell  will  contain  two  placeholders 
that  ultimately  receive  these  values. 

Converting  qlambda  Expressions 

.A,  qlambda  expression  resembles  a  standard  lambda  expression: 

(qlambda  <e>  <formals>  .  <bodxj>) 


’A',-. 
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again,  <e>  is  the  eagerness.  An  eagerness  value  of  none  leads  to  operation 
just  like  the  standard  form: 

(lambda  <forTnals>  .  <body>) 


The  remaining  cases  are  considerably  more  complex,  and  the  published 
description  ([23])  is  somewhat  less  clear  about  the  exact  semantics.  The 
description  given  here  corresponds  to  the  best  understanding  we  were  able 
to  arrive  at  after  consultation  with  members  of  the  Stanford  project. 

For  an  eagerness  of  normal,  the  intent  is  to  create  a  free-standing  task 
when  the  qlambda  expression  is  evaluated.  This  task  repeatedly  tries  to 
read  a  set  of  arguments  from  an  input  queue  and  produce  a  corresponding 
result.  When  the  call  occurs,  all  of  the  arguments  are  evaluated  and  their 
values  are  placed  on  the  task’s  input  queue.  The  caller  receives  a  place¬ 
holder  for  the  value  to  be  computed  by  the  task  and  continues  on  past  the 
call,  pausing  only  if  it  touches  the  result  of  the  call  before  the  task  has 
computed  the  answer.  This  can  be  implemented  using  atomic  queue  opera¬ 
tions  enqueue  to  add  an  item  to  an  existing  queue  and  dequeue  to  remove 
an  item  or  suspend  the  caller  if  no  data  is  available'®.  The  code  required 
is  shown  in  Figure  5.13. 

This  code  deserves  a  little  explanation.  The  work  of  a  qlambda  occurs 
at  two  different  times.  When  the  qlambda  expression  itself  is  evaluated  to 
form  a  ‘‘process  closure”  the  argument  queue  is  created  and  the  independent 
task  begins  running.  The  process  closure  itself  is  created  (by  evaluating 
the  lambda  expression  in  the  above  expansion).  When  this  process  closure 
is  applied  to  arguments  both  the  arguments  and  a  location  (placeholder) 
where  the  result  should  be  stored  are  added  to  the  queue  of  work  for  the 
ta»sk. 

The  task,  of  course,  is  just  a  simple  infinite  loop.  It  attempts  to  read  an 
item  from  its  queue.  When  it  succeeds,  it  will  receive  both  the  arguments 
and  a  placeholder  into  which  to  store  the  result.  It  then  computes  the 
required  function  and  stores  the  result  away  using  determine!.  Since  the 
idaceholder  associated  with  the  task  is  lexically  visible  (as  the  value  of 
the -task),  it  will  continue  running  as  long  as  the  process  closure  remains 

‘^Aloiiiic  enqueue  and  dequeue  operations  can  in  turn  be  written  using  the  built-in 
atomic  operations  set-car-if-eq? !  and  set-carl. 
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(qlambda  normal  <formals>  .  <hody>) 

(let  ( (argument -queue  (make-empty-queue))  ;  (1) 

(the-task) ) 

(define  (thunk)  ;  (2) 

(let  ((arglist  (dequeue  argument-queue))) 

(determine!  (car  arglist) 

(apply  (lambda  <formals>  .  <body>) 

(cdr  arglist)))) 

(thunk) ) 

(set!  the-task  (future  (thunk)))  ;  (3) 

(lambda  <formals>  ;  (4) 

(let  ((answer  (make-placeholder))) 

(enqueue  argument-queue 

(list  answer  .  <formals>)) 

answer))) 

Notes: 

1.  Argument -Queue  is  filled  by  callers  and  emptied  by  the  free-running  task.  Each 
entry  has  a  placeholder  for  the  result  to  be  returned  as  well  as  values  for  the 
arguments. 

2.  Code  for  the  free-running  task:  dequeue  a  set  of  arguments  and  result  placeholder, 
calculate,  store  the  result,  repeat  forever. 

3.  Start  the  task  running  when  qlambda  is  evaluated. 

4.  Code  for  caller:  create  a  placeholder  for  the  answer,  enqueue  arguments  and  place¬ 
holder,  return  the  placeholder. 


Figure  5.13:  QLambda  with  normal  Eagerness 
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at'cfssil)U‘.  Tho  garbage  collector  will  remov'e  the  task  when  the  closure 
becomes  inaccessible* ' . 

The  ftnal  case,  an  eagerness  value  of  eager,  closely  resembles  the  pre- 
\  ious  case.  There  are  two  differences:  the  task  begins  executing  the  body 
immediately  when  the  qlambda  expression  is  evaluated  (rather  than  wait¬ 
ing  for  arguments  to  arrive  from  a  call),  and  the  task  pauses  only  if  it  has 
produced  an  answer  that  has  not  been  requested  yet.  This  latter  ca,se  is 
rather  strange;  it  can  arise  onh'  if  the  body  of  the  qlatnbda  does  not  touch 
any  of  its  arguments.  This  problem  is  handled  in  the  code  shown  in  Fig¬ 
ure  0.14.  leading  to  a  somewhat  unusual  implementation.  In  the  program 
shown  here,  the  free-running  task  begins  operation  as  soon  as  the  qlambda 
expression  is  eviduated,  but  pauses  when  it  attempts  to  use  determine!  to 
store  its  answer.  It  can  proceed  only  when  some  task  has  called  the  closure 
and  therefore  supplied  a  placeholder  for  the  result.  At  the  same  time,  a 
caller  must  pause  until  tlie  free-running  task  has  created  an  entry  on  its 
queiK'  into  which  arguments  for  the  call  can  be  placed.  This  last  constraint 
does  not  permit  a  caller  to  simply  leave  the  arguments  on  a  queue  and  pro¬ 
ceed  --  modifying  the  program  to  permit  this,  however,  is  simple  enough: 
simply  enclose  the  body  of  the  lambda  expression  in  a  future! 

5.4.2  Fork  and  Join 

•Another  common  language  construct  for  parallel  programming  is  the  fork 
and  join  found,  for  example,  in  Simultaneous  Pascal[l2].  This  construct 
is  used  in  a  command-oriented  language  to  indicate  that  the  commands 
between  the  fork  and  join  (which  must  nest  in  the  same  way  that  begin 
and  end  nest)  are  to  be  executed  in  parallel.  When  they  have  all  completed 
execution,  control  continues  past  the  point  of  the  join.  Another  common 
variation  on  thi.s  control  construct  is  do-all,  which  executes  a  single  block 
of  code  with  respect  to  each  index  in  a  given  range  (for  example,  all  the 
subscripts  in  a  one-dimen.sional  array).  A  more  complicated  construct  oper¬ 
ating  in  a  similar  manner  is  the  coenter  of  .4rgus[39].  The  implementation 

"  [iirr>’  is  a  race  condition  here  that  allows  the  task  to  disappear  after  it  has 
received  its  final  .set  of  arguments  hut  before  computing  a  final  value.  One  so¬ 
lution.  due  in  part  to  Halstead,  is  to  change  the  final  line  from  emsver  to 
(future  (begin  (touch  answer)  answer)))  Explaining  howf  this  works  is  left  a,s  a 
challenge  to  the  reader.  .  .  n  aming:  this  may  not  work  if  you  have  a  good  compiler! 
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(qlambda  eager  <formals>  .  <body>)  ^ 

(let  ((argument -queue  (make-empty-queue))  ;  (1) 

(the-task)) 

(define  (thunk)  ;  (2) 

(let  ((args  (cons  (make-placeholder) 

(make-list  (length  <  formats  >) 
make-placeholder) ) ) ) 

(enqueue  argument -queue  args) 

(let  ((answer  (apply  (lambda  <formals>  .  <body>) 

(cdr  args)))) 

(determine!  (car  (car  args))  answer))) 

(thunk) ) 

(set!  the-task  (future  (thunk)))  ;  (3) 

(lambda  <formals>  ;  (4) 

(let  ((answer  (list  (make-placeholder))) 

(args  (dequeue  argument-queue))) 

(determine!  (car  args)  answer) 

(for-each  determine!  (cdr  args)  (list  .  <  formats  >)) 
(car  answer)))) 


Notes: 

1.  Argument-Queue  is  filled  by  the  free-running  task  with  placeholders  where  it  will 
look  for  values  of  arguments  and  where  to  store  a  result.  It  is  emptied  by  callers, 
who  store  the  values  of  arguments  and  where  they  want  the  result  stored. 

Code  for  free-running  task;  make  an  entry  on  2u:gument-queue,  calculate,  store 
result,  repeat  forever.  Notice  that  car  touches  its  argument,  so  this  loop  pauses 
after  calculation  until  a  call  specifies  where  the  result  should  be  stored. 

3.  Start  the  free-running  task  going  when  the  ql2unbda  is  evaluated. 

4.  “Process  closure”  returned  to  the  user:  dequeue  an  entry  from  argument-queue,  fill 
it  with  values  of  arguments  and  a  placeholder  where  the  answer  should  go,  return 
this  placeholder. 

Figure  5.14:  QLaimbda  with  eager  Eagerness 
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of  this  control  structure  in  MtiltiScherne  is  simple  enough:  simply  wrap 
each  command  in  a  future,  gather  a  list  of  the  resulting  placeholders,  and 
touch  them  all. 

.A.  slightly  more  interesting  control  structure,  where  precedence  con¬ 
straints  are  explicitly  indicated,  can  be  achieved  by  using  placeholders  to 
act  as  triggers.  Thus,  each  precedence  constraint  is  represented  by  a  place¬ 
holder.  \\’h('n  the  constraint  is  satisfied,  an  explicit  determine!  operation 
is  jterformed  by  the  task  that  has  satisfied  the  constraint.  Similarly,  any 
block  of  code  that  must  wait  for  a  constraint  to  be  satisfied  merely  touches 
the  a])])ropriate  plactdiolder.  This  organization  can  be  used  conveniently  in 
situations  where  it  is  easy  to  predict  which  block  of  code  is  responsible  for 
satisfying  each  constraint.  This  kind  of  precedence  handling  is  not  directly 
related  to  the  more  powerful  “constraint- based  systems”  familiar  to  the  ar¬ 
tificial  intelligence  community.  These  systems  do  not  possess  this  kind  of 
])redictability,  so  the  trigger  mechanism  will  not  make  the  job  of  designing 
such  a  constraint-based  system  significantly  easier. 

5.4.3  Uniform  System 

One  final  model  for  writing  parallel  programs  is  the  Uniform  System[56],  a 
methodology  advocated  by  BBN  for  use  on  the  Butterfly  computer  system. 
The  essence  of  the  method  is  to  devise  a  “task  generator”  that  runs  as  a 
serialized  critical  region  and  provides  callers  with  information  about  what 
job  to  perform  next.  In  addition,  users  supply  a  main  program  that  is  run 
simultaneously  on  all  available  proce.s',sor.s.  Typically  the  main  program  is  a 
loop  that  calls  a  task  generator,  does  the  work  specified,  and  asks  for  more 
work  that  is,  it  supports  precisely  the  polling  model  of  synchronization 
discussed  in  Section  1.3.2. 

The  proponents  of  this  methodology  point  o\it  th.at  it  can  be  made 
highly  efficient  iiy  carefully  irlentifying  <lata  shared  between  processors 
(used  by  the  task  generators)  and  data  which  can  be  kept  local  to  a  partic¬ 
ular  juocessor.  Indeed,  the  results  achieved  by  using  carefully  implemented 
(but  general  purpose)  ta.sk  generators  is  quite  impressive[15],  leading  to 
v<'ry  nearly  linear  speed  increases  over  the  range  from  one  to  256  processors. 
These  results  show  that  this  techni([ue  can  avoid  many  of  the  “hot-spot" 
(content ion)  problems  found  in  other  methods  used  on  the  Butterfly. 

.Mult i.Sch'^Tiie  deliberately  takes  the  opposite  view  and  encourages  users 


•J 


V 


I 

>, 

i 


I 

■; 

V 


% 

I 

J 

u 

s 


5.4.  OTHER  METHODOLOGIES 


169 


to  ignore  the  distinction  between  private  and  shared  data  storage.  If  Multi- 
Scheme  and  the  Uniform  System  were  otherwise  comparable,  this  difference 
of  api>roach  would  be  an  interesting  one  to  examine  with  respect  to  a  num¬ 
ber  of  architectures.  Unfortunately,  MultiScheme  suffers  in  two  additional 
ways:  it  is  purely  interpreted  (which  makes  it  far  slower  and  at  the  same 
tiiiK'  reduces  contention  for  individual  memory  locations),  and  it  must  strip 
and  restore  type  codes  as  it  references  memory.  This  additional  overhead 
makes  the  direct  comparison  of  the  two  systems  uninformative. 

It  is  interesting  to  notice  that  programs  can  be  written  in  the  “Uni¬ 
form  System  style”  rather  simply  in  MultiScheme  (a  fact  that  has  aided 
some  users  in  moving  programs  from  one  system  to  the  other).  The  only 
problems  are  to  see  how  to  scale  the  number  of  MultiScheme  tasks  to  the 
actual  number  of  processors  available,  and  the  implementation  of  critical 
regions.  Both  of  these  problems  are  easily  solved,  the  former  by  using  the 
N-Interpreters  primitive  mentioned  in  Section  3.2,  and  the  latter  using 
semaphores  built  from  a  spin  lock  using  set !  for  an  atomic  swap  operation 
on  the  lock  cell. 

In  order  to  make  this  more  concrete.  Figure  5.15  shows  one  way  to  write 
a  uniform  system  simulator.  In  order  to  use  this  simulator,  we  must  first 
create  a  package  containing  the  data  used  by  the  tasks,  a  well  as  the  “core” 
of  a  task  generator.  As  an  example,  consider  writing  a  program  to  calculate 
the  factorial  function  by  parceling  out  multiplication  tasks  of  50  numbers 
each.  The  appropriate  core  procedure  is  shown  in  Figure  5.16. 

The  core  should  contain  two  variables  shared  by  all  tasks,  one  of  which 
keeps  track  of  the  next  number  to  be  multiplied  (next-start),  and  the 
otiier  contains  the  final  answer  result.  In  addition,  each  task  will  need  its 
own  local  variable  to  keep  track  of  its  peirt  of  the  product  being  calculated 
(task-value)  —  note  the  use  of  fluid-let  in  Figure  5.16  to  convert  this 
single  shared  variable  into  a  per-task  variable.  In  addition,  the  uniform- 
system  procedure  itself  recpiires  four  procedures: 

end-test 

This  predicate  returns  #T  when  there  is  no  more  work  available  to 
be  parceled  out.  When  this  happens,  the  uniform  system  enters  its 
"termination  phase.”  In  this  example,  termination  begins  when  the 
next  number  to  multiply  (the  global  variable  next -start)  exceeds 
the  argument  originally  given  to  fact,  n. 
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(define  (uniform-system  core) 

(define  result  (make-placeholder)) 

(define  (make-task-generator  core)  ;  (1) 

(let  ( (terminated-tasks  0) 

(end-test  (access  end-test  core)) 

(next  (access  next  core)) 

(end  (access  end  core)) 

(n-tasks  (n-interpreters))) 

(make-critical-region  ;  (2) 

(lambda  () 

(if  (end-test)  ;  (3) 

(let  ((answer  (end))) 

(set!  terminated-tasks  (1+  terminated-tasks)) 

(if  (=  terminated-tasks  n-tasks) 

(determine!  result  answer)) 
kill-task) 

(next))))))  ;  (4) 

(define  task-generator  (make-task-generator  core)) 

(define  (loop)  ((task-generator))  (loop)) 

(define  workers  ;  (5) 

(make-list  (n-interpreters) 

(lambda  ()  (future  ((access  start  core)  loop))))) 
(touch  result))  ;  (6) 


Notes  (see  also  page  172): 


1 .  Make-task-generator  takes  the  core  of  a  user-specified  task  generator  and  creates 
a  full-fledged  handler.  See  Figure  5.16  for  a  sample  core. 


2.  Make-critical-region  serializes  calls  to  its  argument. 


3  Termination  phase:  call  end  to  allow  the  task  to  do  any  final  code,  then  return 
kill-task  as  the  work  to  perform.  If  there  are  no  more  ttisks,  announce  the 
result. 


4  Call  the  user-supplied  task  generator  next. 

5.  Spawn  off  the  actual  tasks,  one  per  processor. 
f3.  Wait  for  an  answer  before  returning  to  the  caller. 


Figure  5.15:  Uniform  System  Simulator 


.... 
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(define  (make-fact-core  n) 

(make-package  fact-core 

((next-start  1)  ;  (1) 

(result  1) 

(task-value))  ;  (2) 

(define  (end-test)  (>  next-start  n))  ;  (3) 

(define  (next)  ;  (4) 

(let  ((start  next-start) 

(end  (min  n  (+  next-start  49)))) 

(set!  next-start  (1+  end)) 

(lambda  () 

(set!  task-value 

(*  task-value 

(multiply-range  start  end)))))) 
(define  (end)  ;  (5) 

(set!  result  (♦  result  task-value)) 
result) 

(define  (start  work)  ;  (6) 

(fluid-let  ((task-value  1))  (work))))) 


Notes  (see  also  page  169): 

1.  Global  variables  used  by  all  tasks. 

2  Local  variables,  per-task. 

3.  Test  for  end  of  task  generation  phase. 

4.  Generator  for  next  task. 

5.  Calculate  final  value  of  task. 

6.  Start  task  by  initializing  per-task  variables  then  processing  work. 
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Figure  5.16;  Core  of  Fact  for  Uniform  System 
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next 

When  the  system  is  not  in  termination  phase,  this  procedure  generates 
the  next  task  to  be  performed.  In  this  example,  the  work  is  to  multiply 
the  current  task- value  (a  per-task  variable)  by  a  range  of  numbers 
(handled  by  the  procedure  multiply-range,  not  shown  here).  The 
result  becomes  the  new  task- value. 

end 

This  procedure  is  called,  as  part  of  a  critical  region,  once  by  each 
task  when  the  system  is  in  its  termination  phase.  It  performs  any  final 
code  before  the  task  is  killed,  and  the  last  task  to  be  killed  is  expected 
to  return  the  final  value  of  the  entire  program.  In  this  example,  it 
multiplies  this  task’s  contribution  to  the  final  answer  (task-value) 
into  the  system-wide  final  result. 

start 

This  procedure  is  called  once  by  each  task  when  it  is  st2irted.  It  is 
expected  to  initialize  any  local  state  information  (in  this  example,  the 
variable  task-value)  and  then  call  its  argument  which  will  cause  it 
to  run  tasks  generated  by  calls  to  next  or  end. 

With  this  package  created,  we  can  examine  the  workings  of  the  uni¬ 
form  system  simulator  of  Figure  5.15.  It  works  by  creating  a  placeholder 
for  its  final  result,  then  converting  the  core  package  of  procedures  into  a 
serialized  task  generator  (using  make-critical-region,  not  shown,  a  sim¬ 
ple  procedure  that  uses  the  atomic  set !  operation  to  create  a  busy-waiting 
lock  procedure).  It  then  makes  the  tasks  by  calling  n-interpreters  to  find 
out  how  many  processors  are  av'ailable  and  creating  a  MultiScheme  task  for 
each  processor.  Nothing  guarantees  that  each  of  these  MultiScheme  tasks 
is  attached  to  a  single  processor,  however  by  making  precisely  one  task  per 
processor  we  can  guarantee  that  the  number  of  tasks  scales  to  the  number 
of  processors. 

The  final  detail  is  the  operation  of  the  critical  region  in  the  task  genera¬ 
tor.  This  calls  the  end-test  from  the  user  supplied  package  to  see  whether 
the  termination  phase  has  begun.  If  not,  the  user’s  next  procedure  is  called 
to  generate  a  new  item  of  work  to  be  executed  by  this  task.  If  so,  the  user’s 
end  procc'dure  is  called  to  let  the  task  perform  any  final  actions  and  return 
an  answer.  If  this  is  the  last  task  remaining,  this  answer  becomes  the  value 
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returned  by  the  entire  simulation.  In  any  case,  the  task  is  then  told  to  call 
kill -task,  causing  it  to  “commit  suicide.” 

Finally,  we  can  define  a  parallel  factorial  function: 

(define  (fact  n)  (uniform-system  (make-fact-core  n)  1)) 


5.4.4  Other  Methodologies:  Summary 

As  we  have  seen,  MultiScheme  can  be  used  (with  some  creative  thought)  to 
emulate  a  large  range  of  parallel  programming  methodologies.  It  provides  a 
nice  environment  for  “cobbling  up”  system  prototypes  written  using  these 
other  methods,  and  this  in  turn  aids  the  rapid  exploration  of  a  number 
of  different  styles  for  writing  a  program.  All  of  these  are  similar  to  Lisp’s 
well  known  advantages  as  a  prototyping  system  in  a  sequential  computing 
environment. 

There  are  limits,  however,  to  the  information  that  can  be  gained  from 
these  emulations.  These  emulations  are  very  useful  in  answering  a  variety 
of  questions: 

•  How  can  I  write  a  selected  program  in  a  particular  methodology? 

•  Does  the  program  produce  the  expected  results?  (If  not,  the  emulator 
is  a  major  aid  in  debugging  the  program.) 

•  Does  the  program  utilize  the  additional  processors? 

•  What  mechanisms  are  needed  to  support  this  methodology? 

On  the  other  hand,  deeper  questions  of  resource  utilization  are  often  ob¬ 
scured  by  the  emulation  overhead.  Furthermore,  MultiScheme  is  deliber¬ 
ately  designed  to  take  certain  kinds  of  resource  allocation  issues  out  of  the 
programmer’s  hands.  This  is  most  noticeable  when  exploring  questions  re¬ 
lated  to  communications  models  or  performance  estimates  that  are  intended 
to  take  communication  costs  into  account. 


Chapter  6 
Conclusion 


Over  a  three-year  period,  MultiScheme  has  evolved  to  the  point  where  it  can 
serve  as  the  system  base  for  significant  parallel  programming  projects.  In 
fact,  any  of  the  examples  discussed  earlier  could  become  the  starting  point 
for  such  an  investigation.  This  idea  is  pursued  in  Section  6.1,  in  which  the 
major  accomplishments  of  the  MultiScheme  project  are  summarized  and  a 
number  of  newly-feasible  research  areas  are  described.  But  the  work  on  the 
MultiScheme  system  itself  is  far  from  complete,  and  some  obvious  steps  to 
improve  it  are  the  topic  of  Section  6.2. 

6.1  What  Have  We  Wrought? 


The  philosophy  behind  the  design  of  MultiScheme  is  simple:  add  some 
straightforward  extensions  to  Scheme  and  thereby  produce  a  new  language 
with  considerable  additional  expressive  power.  For  sequential  computing, 
the  addition  of  placeholders  provides  the  basic  support  for  embedding  logic 
variables  in  Scheme  programs  and  data  structures  (see  Section  2.3)  as  well 
as  call-by-nced  arguments  (Section  2.4).  Furthermore,  MultiScheme  pro¬ 
grammers  have  a  rich  language  for  interacting  with  the  memory  manage¬ 
ment  system  —  the  weak-cons  primitive  allows  an  object  to  be  referenced 
without  altering  its  extent,  while  finalization  allows  an  object’s  extent 
to  be  increased. 

For  parallel  computing,  the  separation  of  placeholders  from  tasks  al¬ 
lows  a  variety  of  programming  methodologies  to  be  expressed  conveniently 


176 


CHAPTER  6.  CONCLUSION 


within  the  MultiScheme  system  (see,  for  example,  the  pipeline  example 
of  Section  5.3  and  Appendix  B,  as  well  as  Section  5.4).  In  addition,  the 
ability  to  change  the  value  of  a  placeholder  (using  mutable-determine !  ) 
has  been  a  key  item  in  the  support  of  work  on  parallel  programming  in 
programs  with  side-effects[35]. 

In  fact,  the  importance  of  placeholder  objects  may  go  well  beyond  these 
examples.  Historically,  the  addition  of  a  new  Lisp  data  type  has  served  to 
focus  the  Lisp  community’s  attention  on  problems  of  interest  to  the  larger 
computer  science  community.  The  cons  cell,  for  example,  proved  to  be  a 
simple  structure  for  studying  sharing;  the  first-class  procedure  forced  at¬ 
tention  onto  the  areas  of  lexical  scoping  and  program  structure;  and  the 
first-class  continuation  invited  experimentation  with  control  structure.  In 
a  similar  manner,  placeholders  help  to  bring  several  very  different  pro¬ 
gramming  issues  into  focus:  parallel  computation,  logic  programming,  and 
normal-order  languages.  The  MultiScheme  work  is  certainly  not  the  first  to 
point  out  the  connections  here,  but  it  is  the  first  to  point  out  that  a  single 
data  structure  can  provide  a  uniform  method  for  studying  all  of  these  areas. 

Another  positive  result  of  the  MultiScheme  project  is  that  a  few  simple 
extensions  to  Scheme  lead  to  a  system  sufficiently  robust  that  it  can  be  used 
for  substantial  application  programs  in  the  paraillel-progrsunming  domain. 
As  a  result  of  this  work  we  can  undertake  a  set  of  projects  that  were  not 
previously  feasible.  Some  of  these  projects  include: 

•  Embedding  a  logic  programming  language  within  MultiScheme  by 
combining  a  unification  algorithm  with  the  placeholder  implementa¬ 
tion  of  logic  variables  (Section  2.3)  and  control  structures  like  the 
dependency-directed  backtracking  described  by  Zabih  et  a/[6l].  Not 
only  should  such  a  system  provide  a  number  of  exciting  possibili¬ 
ties  because  of  the  unusual  control  structure,  but  by  embedding  it 
within  MultiScheme  we  create  a  laboratory  for  exploring  the  use  of 
parallelism  both  within  the  logic  programming  component  and  within 
programs  written  using  that  system.  This  method  of  extending  logic 
programming  into  the  parallel  domain  may  not  be  practical,  even 
though  it  will  certainly  be  instructive:  placeholders  serve  to  repre¬ 
sent  a  form  of  structure  sharing,  but  this  very  sharing  is  a  problem 
in  the  parallel  system  where  several  potential  values  for  a  single  logic 
variable  are  under  consideration  simultaneously. 
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•  MviltiScheme  provides  an  expressive  base  language,  a^'  does  Scheme. 
But  the  power  of  Scheme  derives  largely  from  the  abstractions  that 
can  easily  be  written  in  the  base  language  and  then  used  without 
reference  to  the  implementation  technology.  Such  abstractions  have 
yet  to  be  explored  within  MultiScheme.  There  are  two  areas  ripe  for 
exploration  here;  parallelism  in  control  structures,  and  parallelism  in 
data  structures.  Writing  a  number  of  parallel  search  algorithms  in 
MultiScheme  would  serve  as  a  good  starting  point  for  both  of  these 
areas.  In  the  data  structure  area,  providing  a  package  of  routines 
with  an  interface  similar  to  that  of  the  Connection  Machine’s  *Lisp[5] 
language  should  be  straightforward,  and  would  allow  explorations  into 
data  parallel  programming  in  MultiScheme.  Similarly,  implementing 
the  frons[21]  operation  should  also  be  straightforward  based  on  the 
current  implementation  of  disjoin. 

•  MultiScheme  provides  two  primitive  mechanisms  for  supporting  spec¬ 
ulative  computation:  a  garbage  collector  to  eliminate  unnecessary 
tasks,  and  a  flexible  scheduler  to  facilitate  experiments  in  priority- 
based  execution  of  tasks.  The  first  step  in  understanding  how  this 
support  can  be  used  in  practice  is  the  creation  of  programs  that  ex¬ 
ploit  speculation  to  solve  realistic  problems. 

6.2  What  Remains  To  Be  Done 

While  the  MultiScheme  system  is  now  sufficiently  stable  to  be  used  for  the 
kinds  of  projects  described  in  Section  6.1,  much  remains  to  be  done  to  im¬ 
prove  the  system.  Some  of  this  work  involves  implementation  details  or 
small  changes  to  the  system,  but  there  are  two  important  areas  that  have 
only  been  touched  upon  during  the  construction  of  MultiScheme:  perfor¬ 
mance  and  user  interface.  These  are  extremely  promising  areas  for  future 
research  efforts. 

6.2.1  Performance  Enhancements 

The  most  woefully  neglected  part  of  the  MultiScheme  work  has  been  its 
performance.  There  are  three  major  research  directions  here:  compilation, 
archiferf Ural  ehanges,  and  .sy.stern  changes. 
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Compilation 

Compilation  technology  for  Lisp  and  Scheme  is  relatively  well  de- 
veloped[37].  Most  of  the  standard  techniques  are  directly  applicable 
to  parallel  as  well  as  sequential  programs,  although  some  standard 
optimizations  (such  as  common  sub-expression  elimination)  have  the 
potentially  undesirable  consequence  of  reducing  the  amount  of  par¬ 
allelism  even  though  they  accelerate  a  sequential  thread.  A  Multi- 
Scheme  compiler  would  help  answer  a  number  of  questions  about 
memory  contention  due  to  Lisp’s  reference  pattern,  as  well  as  provid¬ 
ing  the  basic  set  of  tools  needed  for  program  analysis. 

Architecture 

Even  the  best  compiler  can  only  do  a  limited  amount  when  faced  with 
an  architecture  poorly  suited  to  the  language  and  system.  Recent 
work  by  Wu[60]  suggests  that  specialized  hardware  aimed  at  support¬ 
ing  Lisp  may  well  improve  the  cost-performance  ratio  of  a  processor 
within  a  given  technology  family.  The  major  forms  of  hardweire  sup¬ 
port  for  Lisp  are  well  understood  [42],  and  apply  directly  to  parallel 
architectures  as  well.  Beyond  these  support  mechanisms,  however, 
parallel  architectures  are  critically  sensitive  to  memory  latency  and 
bandwidth.  As  stated  in  Section  1.3.1  there  axe  two  fundamental 
requirements  on  a  parallel  axchitecture: 

1.  A  large  portion  of  the  address  space  must  be  shared  by  (acces¬ 
sible  to)  all  processors  in  the  system. 

2.  The  speed  of  reference  to  all  portions  of  the  address  space  must 
be  comparable. 

The  closer  an  architecture  comes  to  realizing  constant  access  speed 
to  all  areas  of  the  address  space,  the  less  will  be  the  impact  of  einy 
non-locality  of  reference  by  programs. 

System 

Unfortunately,  the  “equal  access”  goal  stated  in  the  previous  para¬ 
graph  is  very  hard  to  meet.  Thus,  system  support  aimed  at  keeping 
objects  close  (in  speed  of  access)  to  the  task  referencing  them  may 
have  significant  benefits.  The  simulation  studies  of  Nuth[43]  are  a  first 
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step  in  understanding  the  relationship  between  performance  charac¬ 
teristics  of  the  memory  subsystem  and  overall  system  performance. 
Using  these  same  kinds  of  simulations,  it  may  be  possible  to  tune  the 
memory  and  task  management  to  more  closely  fit  specific  architec¬ 
tures. 

6.2.2  User  Interface 

An  important,  and  often  overlooked,  aspect  of  any  programming  system  is 
the  user  interface.  At  present,  the  MultiScheme  user  interface  is  directly 
based  on  the  MIT  Scheme  user  interface,  in  turn  derived  from  the  standard 
Lisp  Read-Eval-Print  loop.  While  perfectly  serviceable,  this  interface  was 
never  intended  to  help  programmers  visualize  the  activity  of  their  programs 
(cf.  Turbak[57]).  This  visualization  is  even  more  crucial  in  a  parallel  system 
than  in  a  sequential  one.  Providing  an  easily  understood  model  of  the 
program’s  behavior  while  it  is  running  serves  several  important  purposes: 

Displayed  Parallelism 

Any  one  of  a  number  of  techniques  can  be  used  to  keep  the  program¬ 
mer  fully  aware  of  the  amount  of  parallelism  actually  exploited  by  her 
code.  Not  only  does  this  help  programmers  understand  the  amount 
of  parallelism  in  their  programs,  it  also  serves  as  a  subtle  reminder 
that  the  program  has  multiple  concurrent  threads  of  control.  Even 
after  two  years  of  MultiScheme  experience,  I  find  it  easy  to  forget 
this  rather  obvious  fact:  programming  a  parallel  computer  is  not  the 
same  as  programming  a  time-sliced  sequential  computer. 

Visualizing  Dependencies 

When  dealing  with  a  running  parallel  program  it  is  important  to  un¬ 
derstand  which  tasks  depend  on  one  another.  This  information  can 
be  quite  difficult  to  derive,  since  a  future  can  be  stored  in  a  data 
structure  that  is  accessible  from  a  number  of  tasks,  voiding  most  kinds 
of  simple  analysis.  One  vital  role  of  a  good  user  interface  to  Multi- 
Scheme  must  be  to  provide  tools  (visual  or  otherwise)  to  correctly 
cinswer  the  dependency  question,  and  to  subtly  remind  the  program¬ 
mer  that  the  dependency  question  must  be  answered  independently 
of  the  creation  behavior  of  the  program. 
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I/O  handling 

An  important  aspect  of  interacting  with  a  parallel  system  is  how 
tasks  communicate  with  the  user.  MultiScheme  provides  “hooks”  in 
the  form  of  per- task  variables  that  allow  tasks  to  have  independent 
I/O  streams  for  communication,  and  the  Butterfly  Lisp  User  Interface 
(BLUI)  uses  these  to  provide  a  separate  window  for  each  task  which 
attempts  an  I/O  operation.  But  the  ability  to  interrupt  a  specific 
task  is  missing  —  and  implementing  it  will  not  be  easy,  since  there 
are  many  tasks  that  do  not  themselves  perform  I/O  and  so  have  no 
window  through  which  a  user  can  “point”  to  the  task  to  generate  the 
interrupt. 

Debugging  Tools 

A  user  interface  can  also  be  an  extremely  powerful  debugging  tool. 
This,  in  fact,  is  the  source  of  much  of  Lisp’s  power  as  a  program 
development  environment:  the  Read-Eval-Print  loop  allows  users  to 
interact  as  though  they  were  part  of  the  program,  so  that  inspect¬ 
ing  and  modifying  the  state  can  be  done  in  a  natural  manner.  This 
style  becomes  even  more  powerful  when  coupled  with  an  interface 
that  makes  the  computational  environment  visible  and  allows  pro¬ 
grammers  to  tadee  advantage  of  position  to  direct  their  interactions. 
SmallTalk[24]  popularized  this  form  of  interface,  and  a  similar  inter¬ 
face  for  Scheme  was  demonstrated  by  Eisenberg[17]. 

The  fundamental  components  of  a  user-interface  for  MultiScheme  are  un¬ 
derstood;  Turbak’s  visualization  of  control  structure,  Eisenberg’s  visual¬ 
ization  of  binding  and  interaction  environments,  Clamen’s  task  monitoring 
and  dependency  displays,  graph  traversal  (gEirbage  collection)  algorithms, 
BLUI’s  input /output  and  performance  displays.  But  combining  these  into 
a  single,  coherent  interface  remains  a  challenging  project. 

6.3  Parting  Shots 

MultiScheme  is  only  a  start  along  the  long  road  towards  a  convenient  and 
powerful  system  for  programming  parallel  processors.  The  basic  structure, 
largely  inherited  from  its  Lisp  ancestry,  allows  abstractions  to  be  layered 
on  top  of  the  system's  kernel  in  order  to  extend  the  system  in  novel  di- 
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rec'tioiis.  But  work  remains  both  under,  in,  and  above  this  kernel.  The 
architectural  support  under  the  MultiScheme  system,  for  Lisp  in  general 
as  well  as  for  placeholders  and  first-cleiss  continuations  (tasks),  must  be 
pursued  to  the  point  of  producing  a  set  of  criteria  for  evaluating  potential 
machines  as  a  base  for  MultiScheme  implementations.  Critical  to  this  ef¬ 
fort  is  an  understanding  of  the  inter-processor  communication  inherent  in 
the  MultiScheme  model,  and  work  directed  at  the  communication  technol¬ 
ogy  of  the  underlying  hardware.  Work  in  the  system  includes  performance 
enhancements,  user  interface  improvements,  and  the  development  of  tools 
for  debugging  and  “performance  debugging”  of  programs.  Work  above  the 
system  includes  the  construction  of  abstractions  for  a  variety  of  control  and 
data  structures  and  the  implementation  of  significant  application  programs 
that  exploit  these  new  abstractions. 

But  the  single  overriding  lesson  I  have  learned  from  the  MultiScheme 
work  comes  from  a  different  direction  entirely.  In  building  the  MultiScheme 
system,  teaching  students  about  it,  and  in  describing  it  in  this  report,  I  have 
seen  repeatedly  not  how  different  parallel  computing  is  from  sequential  com¬ 
puting,  but  rather  how  similar.  This  lesson  is  perhaps  best  demonstrated 
by  observing  that  most  of  the  changes  made  to  convert  MIT  Scheme  into 
MultiScheme  have  been  re-absorbed  into  the  sequential  system  —  because 
these  additions  have  proven  to  have  important  benefits  in  sequential  com¬ 
puting.  The  outstanding  example,  of  course,  is  the  placeholder  itself:  a  data 
structure  derived  entirely  from  considerations  of  parallel  programming,  yet 
providing  the  base  for  several  important  facilities  in  the  sequential  system. 
It  is  my  belief  that  future  work  will  further  strengthen  this  similarity.  In 
retrospect,  we  will  discover  that  the  importance  of  programming  paral¬ 
lel  systems  comes  from  insights  applicable  to  both  sequential  and  parallel 
computing. 

1  summarize  this  lesson  in  a  simple  analogy: 


sequential  :  parallel  (computing)  real  ;  complex  (analysis) 
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A.l  Introduction  to  Lisp 

The  Lisp  languages  are  distinguished  by  a  number  of  shared  attributes. 
For  those  not  familiar  with  the  Lisp  culture,  here  is  a  brief  (and  doubtless 
biased)  list  of  the  items  most  unlike  other  programming  cultures.  The 
author  makes  no  claim  that  these  are  all  advantages  of  the  language  family; 
they  do  appear,  however,  to  be  facts. 

(Lack  of)  Syntax 

Perhaps  most  distressing  to  those  unfamiliar  with  the  language  is  its 
apparent  lack  of  syntactic  constructs.  This  is  actually  a  mispercep¬ 
tion:  there  is  a  good  deal  of  syntax,  but  it  is  all  superficially  similar. 
Lisp  is  an  expression  language,  as  opposed  to  the  more  common  state¬ 
ment  languages.  Lisp  expressions  are  of  four  kinds: 

1.  Constant  expressions.  For  the  purposes  of  this  report,  the  con¬ 
stants  are  numer2ds,  the  two  booleans  #f  (false,  also  written  eis 
’()  or  ()  since,  for  historical  reasons,  MIT  Scheme  uses  the 
same  object  to  denote  the  empty  list)  and  #t  (true),  and  strings 
surrounded  by  double  quotation  marks  (“"”). 

2.  Variables.  Denoted  by  a  sequence  of  alphEinumeric  characters  and 
certain  additional  characters  (most  importantly  for  this  report 
“ !  ” ,  “?” ,  and  “-”).  HistoricEilly,  Lisp  systems  have  used  a  system 
of  dynamic  binding  for  finding  the  value  of  a  variable.  Scheme, 
however,  uses  a  strictly  lexical  scoping  system  (the  same  system 
used  by  the  Algol  femiily  of  languages). 

3.  Special  forms.  Most  of  the  syntEix  of  the  language  is  embodied 
in  what  are  known  as  the  “special  forms”  of  Lisp.  There  is  a 
short  list  of  these  forms,  and  the  ones  used  in  this  report  are 
shown  in  figures  A.l  and  A. 4  (pages  194  and  198,  respectively). 
A  special  form  is  signalled  by  an  expression  beginning  with  a 
left  parenthesis  followed  by  a  keyword  (the  name  of  the  special 
form).  The  form  ends  with  the  close  parenthesis  matching  the 
opening  one  of  the  form. 

4.  Combinations,  applications,  or  procedure  calls.  These  names  are 
interchangeable,  although  combination  tends  refers  to  the  syn¬ 
tactic  construct,  the  others  to  the  semantics.  These  superficially 
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resemble  special  forms,  a  fact  that  confuses  people  learning  the 
language.  A  combination  is,  like  a  special  form,  surrounded  by 
parentheses.  If  the  first  sub-expression  following  the  left  paren¬ 
thesis  is  not  the  keyword  of  a  specijil  form,  then  the  expression  is 
a  combination.  In  Scheme,  the  value  of  a  combination  is  found 
by  evaluating  each  of  the  subexpressions  (the  order  is  not  spec¬ 
ified).  The  value  of  the  first  subexpression  must  be  a  procedure 
or  continuation,  and  it  is  then  applied  to  the  values  of  the  other 
subexpressions. 

Storage  allocation 

Programmers  do  not  worry  about  allocating  or  releasing  memory  in 
Lisp.  Operations  allocate  memory  eis  needed  by  creating  data  struc¬ 
tures  or  other  objects.  Data  structures  are  made  using  cons  to  create 
an  ordered  pair,  list  to  create  a  singly  linked  list  out  of  ordered  pairs, 
or  vector  to  create  a  one-dimensional  vector.  Memory  is  reclaimed 
by  recycling  those  parts  that  cannot  be  reached  using  the  ordinary 
operations  of  the  system.  This  process  of  recycling  memory  is  known 
as  garbage  collection.  The  operations  car  and  cdr  select  the  first  and 
second  components  of  an  ordered  pair  (respectively),  and  vector- 
ref  selects  entries  in  a  vector  (zero  based).  The  contents  of  a  data 
structure  can  be  modified  using  the  mutators  set-car!,  set-cdr!, 
and  vector-set!  as  appropriate.  Lists  can  be  copied  to  make  vec¬ 
tors  using  the  procedures  list->vector  and  vectors  copied  to  make 
lists  using  vector->list.  There  are  other  data  objects  with  their 
associated  mutators  and  selectors,  introduced  as  needed  throughout 
the  main  body. 

Functional  style 

Algol  family  languages  tend  to  rely  on  sequential  evaluation  of  com¬ 
mands  as  the  primary  control  construct.  The  Lisp  family  of  languages 
uses  applicative  order  evaluation  as  its  primary  control  construct:  the 
subexpressions  of  a  combination  are  evaluated  before  the  body  of  the 
procedure  is  evaluated.  The  Scheme  special  form  begin  is  used  to 
force  sequential  evaluation  of  a  series  of  expressions  when  needed. 

Incremental  program  construction 

Historically,  one  of  the  most  notable  features  of  Lisp  systems  has  been 
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the  emphasis  on  an  environment  supporting  interactive  construction 
of  programs.  In  particular,  the  Lisp  language  does  not  have  the  notion 
of  a  program.  Rather,  it  deals  exclusively  with  procedures  having  a 
standard  function  call  interface.  The  notion  of  a  progrzun  in  other 
languages  is  supported  through  an  informal  (human  based)  system  of 
specifying  certain  top-level  entry  points  into  a  related  group  of  proce¬ 
dures.  The  Lisp  system  supplies  a  stzmdard  interaction  environment 
(the  MIT  Scheme  system  has  the  “read  -  eval  -  print”  interface)  for 
defining  and  calling  procedures. 

Dynamic  type  checking 

A  common  misperception  about  the  Lisp  family  is  that  they  are  type¬ 
less  languages.  In  fact,  they  are  far  from  typeless,  edthough  they  re¬ 
quire  runtime  checking  of  data  types.  Unlike  the  Algol  family  of  lan¬ 
guages,  Lisp  places  data  types  on  the  objects  manipulated  by  the  lan¬ 
guage,  not  on  the  variables  of  the  language  itself.  Usually  no  attempt 
is  made  to  statically  type  variables.  Instead,  the  primitive  operations 
themselves  examine  their  operations  to  determine  whether  they  have 
the  appropriate  data  types.  While  the  Lisp  languages  cannot  strictly 
be  said  either  to  support  or  not  support  user-defined  abstract  data 
types,  it  is  certainly  fair  to  say  that  this  discipline  is  more  one  imposed 
at  the  programmers’  discretion  than  by  the  language  or  programming 
system. 

First-class  objects 

One  way  of  examining  programming  languages  is  to  ask  what  types 
of  objects  are  treated  as  “first-clEiss  citizens”  of  the  lemguage.  A  first- 
class  object  can  be  (a)  stored  as  the  value  of  a  variable,  (b)  passed  eis 
am  argument  to  a  procedure,  (c)  returned  as  the  value  of  a  function, 
and  (d)  stored  in  data  structures.  A  running  theme  of  the  evolution 
of  the  Lisp  family  has  been  an  attempt  to  make  more  objects  first- 
class.  In  the  MIT  Scheme  diailect,  all  objects  aire  first-class  with  the 
exception  of  special  forms  (syntax  or  macros).  In  particular,  binding 
environments,  procedures,  and  continuations  are  first-class  objects. 
The  ramifications  of  these  latter  i  vo  are  explored  in  sections  A. 3 
and  A. 4  respectively. 
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A. 2  Scheme  as  a  Dialect  of  Lisp 

Scheme  evolved  as  a  separate  dialect  of  Lisp  with  three  distinguishing  fea¬ 
tures.  Scheme  was  the  first  Lisp  to  use  lexical  scoping  for  its  variable 
binding  mechanism.  This  major  break  with  the  Lisp  tradition  has  allowed 
Scheme  to  experiment  with  the  block  structure  available  in  the  Algol  fam¬ 
ily  of  languages.  Scheme  has  been  influenced  by  a  very  strong  minimalist 
tradition  from  formal  mathematics.  It  has  fewer  syntactic  constructs  than 
most  other  dialects  and  a  much  smaller  standard  library  of  procedures.  Fi¬ 
nally,  it  emphasizes  consistency  and  simplicity  in  the  language,  relying  on 
compile-time  analysis  for  translation  of  the  simple  constructs  into  efficient 
implementation. 

Readers  familiar  with  CommonLisp[53)  will  notice  the  following  major 
differences: 

•  Scheme  does  not  support  “special  variables.”  Thus,  special  forms  such 
as  symbol-value  and  symbol-function  do  not  exist  in  Scheme.  Sec¬ 
tion  3.3.3  discusses  the  use  of  fluid  variables  in  MIT  Scheme.  These 
provide  similar  properties  but  are  strictly  lexically  scoped.  The  de¬ 
fine  special  form  in  Scheme  always  introduces  a  binding  in  the  cur¬ 
rent  lexical  contour,  and  is  thus  different  from  the  defun  special  form 
of  CommonLisp. 

•  Scheme  has  no  iteration  constructs.  Instead,  all  implementations  of 
Scheme  are  required  to  exhibit  fully  reductive  behavior  (sometimes 
called  tail-recursion).  That  is,  each  sub-expression  of  an  initial  expres¬ 
sion  can  be  syntactically  determined  to  be  either  a  subproblem  or  a 
reduction  of  the  full  expression.  No  stack  space  is  required  to  han¬ 
dle  the  evaluation  of  a  reduction  sub-expression.  The  result  of  this 
requirement  is  that  some  (syntactically)  recursive  procedures  may  be 
executed  in  an  iterative  manner.  This  area  is  described  in  more  detail 
in  Section  A. 4. 

•  Scheme  treats  all  sub-expressions  of  a  combination  in  an  identical 
manner.  In  particular,  the  first  sub-expression  is  evaluated  using  the 
same  evaluation  rules  it  uses  for  the  other  sub-expressions.  There  is  no 
separate  function  name  binding  environment.  None  of  the  associated 
special  forms  exist  (funcall,  flet,  etc.). 
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Name 

Description 

and, 

or 

(and  expi  ...  €xp„),  (or  expi  ...  eip„) 

Short-circuit  (or  conditional)  and,  or. 

begin 

(begin  expi  ...  expn) 

Sequentially  evaluate  exps,  returning  value  of  exp„. 

cond 

(cond  (predi  .  conseqi)  . . .)) 

McCarthy’s  conditional  construct. 

define 

(define  name  exp) 

Add  binding  for  name  to  exp  in  current  environment. 

(define  (name  .  params)  .  body)  expands  to 
(define  name  (lambda  (params)  .  body)). 

delay 

(delay  exp) 

Memoized  delay  of  exp  until  it  is  forced. 

■ 

(if  predicate  consequent  alternative) 

If-Then-Else  construct. 

lambda 

(lambda  (param\  ...)  .  body) 

Create  a  procedure  (closure). 

let 

(let  ((namei  expi)  . . .)  .  body) 

Evaluate  exps,  then  bind  names  to  these  values,  then  eval¬ 
uate  body.  Evaluation  and  binding  order  are  not  specified. 

letrec 

(letrec  ((namei  expi)  . . .)  .  body) 

Similar  to  let,  but  exps  can  reference  the  names. 

quote 

(quote  expression)  or  ’expression 

Return  expression  without  evaluating  it. 

set! 

(set!  name  expression) 

Assign  value  of  expression  to  lexical  variable  name. 

Note:  For  a  full  definition  of  the  standard  Scheme  language,  please  refer  to  [49] 
Figure  A.l;  Standard  Scheme  Syntax  Used  in  Report 
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•  The  special  form  lambda  in  Scheme  evaluates  to  a  procedure  object 
—  the  “closure”  of  CommonLisp.  There  is  no  need  for  the  function 
special  form  (or  its  abbreviation, 

•  The  special  form  set !  in  Scheme  is  roughly  equivalent  to  the  setq 
form  of  CommonLisp  but  it  will  not  create  a  binding.  Scheme  has 
no  equivalent  for  the  setf  macro,  but  uses  the  primitive  operations 
set-car!,  set-cdrl,  ajid  vector-set!  explicitly. 

•  Scheme  guarantees  neither  the  order  of  evaluation  of  operands  nor 
the  order  of  binding  in  a  procedure  call.  For  sequential  evaluation, 
the  special  form  begin  must  be  used.  For  sequential  binding,  either 
the  macro  let*  or  an  equivalent  progrsim  structure  must  be  used. 

A. 2.1  The  Standard  Scheme  Dialect 

There  is  a  standard  Scheme  dialect  supported  by  a  number  of  different 
implementations.  This  di2dect  is  described  in  a  short  document[49]  that 
includes  a  complete  denotational  semantics  for  the  language.  A  very  brief 
summary  of  the  syntactic  constructs  used  in  the  examples  of  this  report  is 
included  in  Figure  A.l. 

The  procedures  used  in  this  report  which  either  differ  from  CommonLisp 
or  are  not  frequently  encountered  are  shown  in  Figure  A. 2.  In  addition,  a 
smattering  of  common  Lisp  procedures  are  used  without  explanation  (such 
as  car,  cdr,  eq?,  etc.). 

Figure  A. 3  shows  a  simple  program  demonstrating  some  of  the  syntax  of 
the  Scheme  language  and  most  of  the  procedures  described  in  Figure  A. 2. 

A. 2. 2  MIT  Extensions  to  Scheme 

The  purpose  of  the  Scheme  standard[49]  is  to  provide  a  language  base.  This 
base  is  then  subject  to  local  extensions.  The  MIT  Scheme  system  contains 
a  number  of  extensions  used  throughout  this  report,  shown  in  Figure  A. 4. 
These  extensions  provide  three  features  not  conveniently  available  in  the 
standard  dialect. 

First-class  environments 

The  MIT  dialect  of  Scheme  has  been  extended  to  allow  lexical  binding 
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Name 

Description 

apply 

(apply  procedure  argument-list) 

Call  pix)cedure  with  the  arguments  in  the  argument-list. 
This  “spreads”  the  argument  list  before  calling  the  pro¬ 
cedure. 

display 

Print  its  one  argument  out  in  a  human-  (rather  than 
machine-)  readable  format. 

for-each 

(for-each  procedure  list) 

Applies  the  procedure  to  each  item  in  the  list.  Procedures 
are  invoked  “for  effect  only,”  thus  the  value  returned  is 
unspecified.  This  is  MacLisp’s  mapc  procedure. 

map 

(map  procedure  list) 

Similar  to  for-each,  but  returns  a  list  of  the  results  pro¬ 
duced  by  procedure.  This  is  MacLisp’s  mapcar  procedure. 

newline 

Begin  a  new  line  on  the  primary  output  device. 

Figure  A. 2:  Standard  Procedures  Used  in  Report 
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(define  (display-scaled-matrix  matrix  scale) 


(define  (scale-row  row)  ;  (1) 

(map  (lambda  (item)  (*  item  scale))  row)) 
(define  (display-row  row)  ;  (2) 

(define  (each-item  item)  ;  (3) 

(display  item) 

(display  "  ")) 

(for-each  each-item  row) 

(newline) ) 

(for-each  ;  (4) 


(lambda  (row)  (display-row  (scale-row  row))) 
matrix) ) 


Notes: 

1.  The  procedure  scale-row  takes  a  row  of  a  matrix  (stored  as  a  linked  list)  and 
produces  a  list  of  scaled  values  using  the  procedure  map. 

2.  The  procedure  display-row  uses  for-each  to  display  each  item  in  a  given  input 
row,  then  uses  newline  to  start  a  new  line  of  output. 

3.  Each-item  is  called  once  for  each  item  in  the  matrix.  It  displays  the  item  followed 
by  a  space. 

4.  This  is  the  main  body  of  the  procedure  display-scaled-matrix.  It  uses  for-each 
to  display  the  scaled  rows  of  the  matrix.  This  procedure  atssumes  that  a  matrix  is 
stored  as  a  list  of  lists  (rather  than  the  more  traditional  vector  of  vectors). 

Figure  A. 3:  A  Simple  Scheme  Program 
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Name 

Description 

access 

(access  name  expi)  and 
(sat!  (access  name  expi)  exp2) 

The  variable  name  in  the  environment  exp\ . 

cons-stream 

(cons-stream  exp\  expD  expands  into 
(cons  exp\  (delay  exp^)) 

fluid-let 

(fluid-let  Cinamei  expi)  . . .)  .  body) 

Evaluate  all  of  the  exps,  then  modify  the  bindings  of  the 
existing  (lexical)  variables  (names).  Evaluate  the  body, 
then  restore  the  original  values  of  the  variables.  Inter¬ 
acts  with  call-vith-current-continuation  to  guaran¬ 
tee  that  values  are  restored  even  when  exit  is  performed 
using  continuations.  See  description  in  Section  3.3.3. 

the-environment 

(the-environment) 

Return  the  current  lexical  environment  as  a  Scheme  ob¬ 
ject.  Used  with  access. 

Figure  A. 4:  MIT  Syntax  Used  in  Report 
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environments  to  be  treated  as  first-class  objects.  Environments  are 
made  available  to  a  Scheme  program  by  using  the  the-environment 
special  form  or  one  of  a  number  of  macros  built  using  it.  These  return 
as  their  value  the  current  environment.  The  value  of  a  variable  in 
a  given  environment  can  be  found  using  the  access  special  form, 
and  the  syntax  of  the  set!  special  form  allows  the  use  of  an  access 
special  form  in  place  of  a  variable  name.  Finally,  the  procedure  eval 
can  be  used  as  a  “compile  and  run”  operation.  It  receives  both  a 
program  to  be  executed  (typically  represented  as  a  list)  and  a  binding 
environment  for  execution. 

Stream  processing 

Section  2.5  introduces  the  notion  of  stream  processing,  a  very  powerful 
method  for  dealing  with  simulations  in  a  functional  system.  MIT 
Scheme  provides  only  one  simple  additional  syntax  for  dealing  with 
streams:  cons-streajn  expands  as  shown  in  Figure  A. 4.  Using  this 
syntactic  extension  along  with  the  standard  delay  special  form  and 
primitive  operations  car,  cdr,  and  force,  the  entire  array  of  stream 
processing  functions  are  easily  defined. 

Fluid  variable  binding 

Section  3.3.3  discusses  MIT  Scheme’s  fluid  variable  concept.  The 
essential  idea  is  to  allow  a  single  lexical  variable  to  represent  differ¬ 
ent  values  depending  on  the  control  path  taken  to  reach  the  current 
execution  state.  The  special  form  fluid-let  is  introduced  in  MIT 
Scheme  to  make  the  addition  of  fluid  bindings  convenient. 

In  addition  to  these  language  extensions,  MIT  Scheme  is  largely  distin¬ 
guished  by  its  particular  choice  of  runtime  environment.  This  is  discussed 
in  Section  A. 5. 

A. 3  Procedures  as  Objects 

While  most  modern  dialects  of  Lisp,  including  CommonLisp  and  Scheme, 
allow  procedures  (also  known  as  closures)  to  be  treated  as  first-class  ob¬ 
jects.  the  practice  is  rare  in  most  other  languages  and  unfamiliar  to  most 
j)rogrammers.  This  report  is  certainly  not  the  place  to  introduce  the  power 
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and  flexibility  this  allows  (see,  for  example,  references  [7]  and  [55]),  but  an 
ability  to  at  least  understand  programs  written  using  this  ability  is  essen¬ 
tial  to  a  number  of  the  exEimples,  especially  in  Chapter  4.  The  essential 
points  to  bear  in  mind  can  be  summarized  in  the  following  “mantras”  of 
the  Scheme  language: 

•  Every  expression  has  a  value. 

•  The  value  of  a  lambda-expression  is  a  procedure. 

•  The  free  variables  of  a  procedure  receive  their  values  from  the  envi¬ 
ronment  in  which  the  procedure  was  created  (i.e.  lexical  scoping). 

•  The  bound  variables  of  a  procedure  receive  their  values  from  the  ar¬ 
gument  list  in  the  call  to  the  procedure. 

Because  variables  are  lexically  scoped  and  procedures  can  be  created  “on 
the  fly”  using  laimbda-cxpressions,  procedures  are  often  used  to  encapsulate 
expressions  whose  value  is  to  be  computed  at  a  later  time.  Procedures  used 
in  this  manner  are  known,  for  historical  reasons,  as  thunks.  This  term  is 
used  somewhat  loosely  in  this  report,  and  does  not  necessarily  imply  that 
the  procedure  hcis  no  formal  parameters  (the  traditional  usage). 

The  sample  program  shown  in  Figure  A. 3  demonstrates  a  very  common 
use  of  lambda-expressions  to  create  a  procedure  for  use  as  an  input  argu¬ 
ment.  Figure  A. 5  shows  three  equivalent  ways  of  defining  a  very  simple 
procedure-generating  procedure,  and  Figure  A. 6  is  a  demonstration  of  its 
use.  If  the  syntax  or  operation  of  this  procedure  is  difficult  to  understand, 
a  thorough  treatment  of  these  topics  is  provided  in  the  chapters  1  and  3  of 
reference  [7]. 

A. 4  Continuations  as  Objects 

There  is  one  type  of  object  available  in  Scheme  that  does  not  exist  in  other 
programming  languages:  the  continuation.  Continuations  were  introduced 
as  part  of  the  formal  description  of  programming  languages  through  denota- 
tional  semantics.  They  are  used  there  to  provide  a  mathematical  handle  for 
expressing  control  flow  in  programs  (see  Stoy[55]  for  an  excellent  overview 
of  the  mathematics  involved). 
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(define  (make-incrementer !  amount) 
(lambda  (increment) 

(set!  amount  (+  amount  increment)) 
amount ) ) 


(define  make-incrementer I 
(lambda  (amount) 

(lambda  (increment) 

(set!  amount  (+  amount  increment)) 
amount) ) ) 


(define  (make-incrementer!  amount) 
(define  (do-increment!  increment) 
(set!  amount  (+  amount  increment)) 
amount) 

do-increment ! ) 


Figure  A. 5:  Three  Ways  to  Define  a  Procedure-Generator 
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=>  (define  incrementer-a 

(madce-incrementer !  3)) 
=>  (define  incrementer-b 

(make-incrementer !  3)) 
=>  (define  incrementer-c 
incrementer-a) 

^  (incrementer-a  5) 

=>  (incrementer-b  3) 

=>  (incrementer-b  2) 
(incrementer-a  2) 
(incrementer-c  3) 
(incrementer-a  2) 


=>  ((make-incrementer!  3)  2) 


INCREMENTER-A 

INCREMENTER-B 

INCREMENTER-C 

8 

6 

8 

10 

13 

15 

5 


Figure  A. 6:  Use  of  Meike- Increment er! 


A. 4.1  Continuations:  An  Introduction 

The  main  idea  behind  a  continuation  is  that  every  expression  or  statement 
in  any  language  is  provided  with  a  function  it  is  expected  to  cjill  when  it 
hcis  completed  its  own  processing.  The  notion  of  “returning  a  value”  is 
thus  eliminated  and  replaced  with  the  simpler  (mathematically)  notion  of 
calling  a  function. 

For  example,  the  meaning  (semantics)  of  the  Scheme  statement 
(if  predicate  consequent  alternative) 

can  be  described  as  follows.  First,  each  of  the  three  sub-expressions  is  con¬ 
verted  into  its  equivalent  semantic  function  (i.e.  the  mathematical  function 
that  computes  the  same  value  the  sub-expression  itself  computes).  Recall 
that  every  semantic  function,  including  these  three,  will  receive  a  continu¬ 
ation  as  one  of  its  arguments. 

The  semantic  function  of  the  if  expression  itself,  then,  receives  a  con¬ 
tinuation,  K.  Since  the  first  thing  an  if  statement  must  do  is  to  evaluate 
the  predicate  expression,  the  semantic  function  for  the  if  statement  will 
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call  the  semantic  function  for  the  predicate  sub-expression.  But  this  re¬ 
quires  a  continuation  expressing  the  work  to  be  performed  after  the  value 
of  the  predicate  has  been  computed.  If  we  call  the  semantic  functions  for 
the  consequent  and  the  alternative  consequent,/  and  alternative,/,  re¬ 
spectively,  then  this  new  continuation  will  receive  the  value  of  the  predicate 
sub  expression  and  call  either  consequent,/  or  alternative,/. 


(define  (if,/-generator  predicate,/  consequent,/  alternative,/) 
(lambda  (k) 

(define  (k*  value-of -predicate) 

(cond  ((eq?  value-of -predicate  if) 

(alternative,/  k)) 

(else  (consequent,/  k)))) 

(predicate,/  «’))) 

Figure  A. 7:  The  Semantic  Function  for  if 


To  help  readers  who  are  not  familiar  with  the  notion  of  continuations, 
Figure  A. 7  shows,  using  Scheme  notation,  a  procedure  that  generates  the 
semantic  function  for  an  if  expression  given  the  semamtic  functions  for 
the  sub-expressions.  The  most  important  point  to  notice  is  that  k'  uses 
K.  the  continuation  for  the  if  expression  itself,  as  the  continuation  for  use 
by  the  consequent  and  alternative  sub-expressions.  This  corresponds  to 
the  fact  that  the  predicate  sub-expression  in  Scheme  is  a  “subproblem” 
of  the  original  expression  (there  is  more  work  to  be  performed  after  its 
value  is  calculated)  whereas  the  consequent  and  alternative  sub-expressions 
are  “reductions”  of  the  original  —  their  value  is  the  value  of  the  overall 
expression.  This  idea  will  be  examined  again  in  Section  A. 4. 3. 

A. 4. 2  Continuations  in  the  Scheme  Language 

In  1978,  Steele[54]  introduced  the  idea  of  using  continuations  as  a  way  of 
analyzing  Scheme  programs  during  the  compilation  process.  He  explored 
this  possibility  and  demonstrated  the  idea  by  building  the  first  working 
compiler  for  the  Scheme  langiiage.  He  also  introduced  a  programming 
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style,  called  “continuation  passing,”  that  makes  the  continuations  explic¬ 
itly  visible  within  the  prog^^lm  itself.  Exploiting  Scheme’s  ability  to  treat 
procedures  as  objects,  Steele  showed  a  simple  procedure  for  converting  any 
procedure  written  in  Scheme  into  an  equivalent  procedure  that  does  not 
return  a  value  but  rather  invokes  an  explicitly  supplied  continuation. 

In  Steele’s  system,  an  expression’s  continuations  was  simply  a  procedure 
of  one  argument  (the  value  computed  by  the  expression).  The  fact  that  this 
simple  conversion  process  can  be  applied  to  any  program  demonstrated  that 
the  Scheme  language  was  sufficient  to  implement  continuations  without 
any  e.xtensions.  The  availability  of  continuations  within  the  language  itself 
adds  a  vast  amount  of  expressive  power  since  it  encompasses  all  the  forms  of 
control  structure  that  can  be  described  using  the  mechanics  of  denotational 
semantics.  The  ramifications  of  the  ability  to  program  using  continuations 
are  still  being  explored  by  the  Scheme  community. 

Unfortunately,  the  procedures  resulting  from  the  conversion  process  are 
often  difficult  to  understand.  The  Eirgument  that  continuations  need  not 
be  added  to  the  Scheme  language  is  factually  correct.  It  has  as  much  valid¬ 
ity  as  the  statement  that  “the  names  of  formal  parameters  can  be  chosen 
arbitrarily.”  And  both  of  these  arguments  have  the  same  basic  flaw:  the 
form  in  which  a  statement  is  written  can  have  a  major  impact  on  how  eas¬ 
ily  a  person  can  understand  the  statement.  While  understanding  that  the 
language  does  not  inherently  need  any  extensions  to  support  programming 
using  continuations,  the  Scheme  community  nevertheless  chose  to  add  one 
operation  to  the  language  to  ease  the  chore. 

Instead  of  rewriting  a  program  so  that  all  continuations  are  made  explic¬ 
itly  visible  in  the  text,  Scheme  supports  an  operation  with  the  somewhat 
daunting  name  call-with-current-continuation.  Rather  than  name 
every  continuation  and  pass  them  explicitly  as  procedures,  this  operation 
allows  Scheme  programs  to  acquire  a  name  for  just  those  continuations  used 
in  a  way  not  svipported  directly  by  the  syntax  of  the  language.  The  contin¬ 
uation  objects  made  available  using  this  operation  are,  from  the  point  of 
view  of  a  programmer  on  a  serial  machine,  just  another  kind  of  procedure 
object.  They  expect  a  single  argument,  ordinarily  the  value  computed  by 
the  expression  whose  continuation  they  represent. 

A  simple  use  of  call-with-current-continuation  is  demonstrated 
in  Figure  A. 8.  Here  the  continuation  is  used  to  support  a  “non-standard 
exit"  .sinhlar  to  the  CommonLisp  catch  and  throw  operations  or  the  exit 
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(define  (list-of-square-roots  list) 
(call-with-current-continuation 
(lambda  (early-exit)  ;  (1) 

(map  (lambda  (object)  ;  (2) 

(cond  ((not  (number?  object)) 

(early-exit  'NOT-A-NUMBER))  ;  (3) 

((negative?  object) 

(early-exit  'NEGATIVE))  ;  (4) 

(else  (sqrt  object))))  ;  (5) 

list)))) 


Examples  of  Use: 

=>  (list-of-square-roots  '(4  9  16))  ^  (234) 

=►  (list-of-square-roots  ’(4  x  16))  -s^  NOT-A-NUMBER 

=>  (list-of-square-roots  ’(4  9  -16))  -v-*  NEGATIVE 

Notes: 


1.  This  procedure  (the  value  of  the  lambda  expression)  is  called  with  e^urly-exit 
bound  to  a  continuation  object. 

2.  Recall  that  map  applies  a  procedure  to  each  element  of  an  input  list  and  creates  a 
list  of  the  resulting  values. 

3.  First  premature  exit.  Notice  that  the  continuation  object,  eeurly-exit,  is  treated 
exactly  like  a  procedure  of  one  argument.  The  value  of  that  argument  is  returned 
as  the  value  of  that  call  to  call-with-current-continuation  that  created  the 
continuation  object. 

4.  Second  premature  exit.  See  note  3,  above. 

5.  Ordinary  exit.  The  square  root  is  returned  using  the  ordinary  (implicit)  continua¬ 
tion  —  in  this  caise  one  created  somewhere  inside  of  the  map  procedure. 

Figure  A. 8:  Simple  Use  of  Call-With-Current -Continuation 
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procedure  of  UCSD  Pascal. 

Notice  the  somewhat  unusual  interface  provided  by  the  call-with-cur- 
rent -continuation  primitive.  It  takes  one  argument,  a  procedure.  This 
procedure  is  called  and  passed  its  own  continuation  as  its  argument.  This 
continuation,  of  course,  is  the  one  corresponding  to  returning  from  the  call 
to  call-with-current-continuation  itself.  Since  Scheme  requires  every 
procedure  to  return  a  value,  the  continuation  object  itself  expects  one  argu¬ 
ment;  the  value  to  be  returned  from  call-with-current-continuation. 


Calling  a  continuation  object  has  the  effect  of  returning  from 
the  call  to  call-with-current-continuation  that  created  the 


continuation. 


It  is  important  to  bear  in  mind  that  the  operation  call-with-current- 
continuation  is  nothing  more  than  a  way  of  avoiding  the  inconvenience  of 
syntactically  rewriting  Scheme  procedures.  In  particular,  the  continuation 
object  created  in  this  way  is  a  first-class  object.  It  has  indefinite  extent  and 
can  be  used  multiple  times,  just  like  an  ordinary  procedure.  This  implies,  of 
course,  that  a  call  to  call-with-current-continuation  may  return  more 
than  once:  an  unusual  but  useful  possibility. 


A. 4. 3  Continuations:  One  Implementor’s  View 


There  are  a  number  of  different  implementations  of  the  Scheme  language, 
and  each  of  them  has  its  own  techniques  for  handling  first-class  continuation 
objects.  The  description  here  is  based  on  the  MIT  interpreter  since  it  forms 
the  basis  of  MultiScheme. 

Because  of  Scheme’s  lexical  scoping  and  consequent  block  structure,  its 
implementation  in  many  ways  resembles  that  of  the  more  familiar  Algol 
family  of  languages  (Pascal  and  Ada,  for  example).  Common  implementa¬ 
tions  of  these  languages  rely  (at  least  conceptually)  on  a  stack  to  support 
recursive  procedure  calls.  Scheme  differs  in  substantial  ways,  however,  and 
the  simplest  way  to  describe  Scheme’s  handling  of  continuations  is  by  con¬ 
trast  with  a  stack- based  implementation  of  one  of  these  languages.  The 
three  major  differences  are  summarized  in  Figure  A. 9.  A  typical  Pascal 
implementation  (such  as  the  UCSD  implementation)  uses  a  single  stack  for 
three  different  kinds  of  activities. 
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i: 

f. 


f: 

•f. 


1.  Variable  bindings.  The  values  of  formal  parameters  to  a  procedure 
are  pushed  on  the  stack  along  with  the  “static  link”  to  the  lexically 
enclosing  stack  frame.  All  references  to  variables  are  through  this 
stack  area,  either  to  a  local  variable  stored  in  the  frame  itself  or 
through  the  static  link  to  enclosing  frames*.  Return  from  a  procedure 
includes  popping  the  parameters  off  the  stack. 

2.  Return  addresses  or  “dynamic  link.”  Since  procedures  are  entered, 
execute,  and  then  return  to  the  caller,  the  stack  is  a  natural  data 
structure  for  storing  this  call  chain.  The  address  where  control  will 
return  when  the  current  procedure  completes  is  therefore  pushed  on 
a  stack  when  the  call  begins  and  is  popped  when  the  procedure  ends. 

3.  Intermediate  storage  while  evaluating  sub-expressions.  In  calculating 
the  value  of  a  complicated  arithmetic  expression,  for  example,  the 
values  of  sub-expressions  axe  pushed  onto  the  stack  as  they  axe  com¬ 
puted  axid  then  popped  back  off  when  they  are  combined  with  the 
values  of  other  sub-expressions. 

The  Scheme  language  is  more  flexible  than  most  Algol  family  languages, 
and  this  flexibility  comes  at  a  price.  Scheme's  ability  to  create  and  return  a 
procedural  value  at  run  time  (as  in  Figure  A. 5)  requires  that  some  variable 
bindings  have  a  lifetime  extending  beyond  the  execution  time  of  the  pro¬ 
cedure  that  created  them.  Calls  to  these  procedures  cannot,  therefore,  use 
the  procedure  call  stack  to  store  variable  bindings.  Instead,  in  the  general 
case,  bindings  must  be  stored  in  the  heap  (the  same  place  where  cons  cells 
are  allocated).  This  leads  to  the  first  statement  in  Figure  A. 9. 

One  of  the  unusual  features  of  Scheme,  mentioned  in  Section  A. 2,  is  that 
there  are  no  iteration  constructs.  All  implementations  must  correctly  im¬ 
plement  the  notions  of  subproblem  and  reduction  mentioned  earlier.  In  order 
to  understand  the  impact  of  this  requirement  it  is  important  to  understand 
the  relationship  between  these  concepts  and  the  use  of  the  stack.  For  the 
purposes  of  this  report,  we  can  use  the  following  informal  definitions: 

Subproblem: 

A  sub-expression  is  a  subproblem  of  an  original  expression  if  the  value 


'Sometimes  a  “display”  is  used  to  access  the  enclosing  frames  rather  than  a  chain  of 
static  links.  The  frames,  however,  are  still  stored  on  the  stack. 
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1.  Values  of  variables  are  not  (in  general)  stored  on  the  Scheme 
stack. 

2.  Procedure  call  in  Scheme  does  not  push  entries  onto  the 

stack.  Subproblems  push  entries  on  the  stack,  but  reductions  do 
not. 

3.  The  current  continuation  is  the  stack,  and  it  (or  a  copy)  can 
become  directly  visible  to  programs. 

Figure  A. 9:  Scheme’s  Stack  vs  P^lsc^ll’s  Stack 


of  the  full  expression  requires  finding  the  value  of  the  sub-expression 
and  then  doing  additional  work.  Alternatively,  a  subproblem  is  a  sub¬ 
expression  evaluated  using  a  continuation  othtr  than  the  one  supplied 
for  the  overall  expression. 

Reduction: 

A  sub-expression  is  a  reduction  of  an  original  expression  if  the  value 
of  the  full  expression  is  the  value  of  the  sub-expression  (should  that 
sub  expression  ever  by  evaluated).  Alternatively,  a  reduction  is  a 
sub-expression  evaluated  using  the  same  continuation  as  that  of  the 
original  expression. 

Using  these  definitions  and  Figure  A. 7  we  can  see  that  the  predicate 
sub-expression  of  an  if-expression  is  a  subproblem  (it  has  k\  not  k,  for 
its  continuation),  while  the  consequent  and  alternative  are  reductions 
(they  have  the  original  k  as  their  continuation).  A  similar  analysis,  based 
on  the  formal  semantics  given  in  [49],  will  distinguish  subproblems  from 
reductions  for  all  the  sub-expressions  in  the  Scheme  language.  The  two 
most  important  reductions  in  the  language,  leading  to  the  majority  of  the 
so-called  tail-recursive  nature  of  the  language,  are 

1.  The  final  sub-expression  of  a  begin  special  form.  That  is,  each  of 
the  earlier  sub-expressions  is  a  subproblem  (using  a  continuation  that 
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causes  the  evaluation  to  continue  on  to  the  next  sub-expression).  The 
final  sub  expression,  however,  is  a  reduction  since  its  value  is  the  value 
of  the  entire  expression.  This  reduction  is  particularly  significant  since 
there  is  an  implicit  begin  specird  form  around  the  body  of  all  pro¬ 
cedures  (including  the  bodies  of  let,  letrec,  and  fluid-let  special 
forms)  and  the  consequents  of  cond  special  forms. 

2.  The  body  of  a  procedure.  That  is,  after  evaluating  all  of  the  sub¬ 
expressions  of  a  combination,  the  evaluation  of  the  combination  re¬ 
duces  to  the  body  of  the  procedure. 

With  this  understanding,  we  can  reexamine  the  original  question:  what  is 
the  relationship  between  these  two  concepts  and  the  use  of  a  stack  in  the 
implementation.  The  answer  lies  in  the  fact  that  the  continuations  are  for 
the  most  part  left  implicit  in  Scheme  programs.  As  long  as  this  is  true, 
an  analysis  of  the  language  will  reveal  that  the  continuations  created  for 
subproblems  are  used  in  a  last-in  first-out  manner  and  can  therefore  be 
efficiently  stored  on  a  stack.  This  should  come  as  no  surprise,  since  the  lan¬ 
guage  has  none  of  the  control  constructs  that  would  lead  to  anything  other 
than  the  ordinary  stack-like  discipline  of  procedure  call  in  other  languages. 

In  fact,  the  MIT  Scheme  implementation  does  use  a  stack  for  storing 
continuations.  Thus,  whenever  a  new  subproblem  is  about  to  be  evaluated 
a  new  entry  is  pushed  on  the  continuation  stack,  and  completion  of  an 
evaluation  step  (where  the  semantic  function  would  call  the  original  con¬ 
tinuation  of  an  expression)  pops  this  continuation  off  of  the  stack.  This 
usage  of  the  stack  corresponds  to  the  return  address  use  (number  2)  de¬ 
scribed  earlier.  But  there  is  an  important  difference.  Notice  that  Scheme 
pushes  an  entry  onto  the  stack  only  for  subproblems  of  an  expression,  not 
reductions.  Since  the  body  of  a  procedure  is  a  reduction  it  follows  that,  as 
shown  in  Figure  A. 9,  procedure  call  in  Scheme  does  not  push  entries  onto 
the  stack. 

Instead,  it  is  the  position  in  which  the  call  occurs  that  determines 
whether  an  entry  is  pushed  on  the  stack.  Calls  occurring  in  subproblem 
position  push  entries  onto  the  stack;  calls  occurring  in  reduction  position 
leave  the  stack  unchanged.  It  is  this  property  that  allows  a  (syntactically) 
recursive  Scheme  program  to  operate  without  requiring  additional  space 
(i.e.  iterate).  This  is  a  generalization  of  the  more  common  tail  recursion 
optimization  found  in  compilers  for  some  languages. 
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With  this  terminology  and  implementation  background,  we  can  explain 
the  implementation  of  call-with-current-continuation.  Rather  than 
produce  a  full  procedure  at  every  point  where  the  semauoics  of  the  lan- 
g’aage  requires  a  new  continuation,  the  MIT  implementation  pushes  an 
entry  onto  a  stack.  This  is  fine  for  all  of  the  ordinary  operations  of  the 
language,  since  they  require  knowing  only  the  immediately  following  oper¬ 
ation  to  be  performed.  Call-with-current-continuation,  on  the  other 
hand,  must  make  an  applicable  object  with  indefinite  extent  that  can  be 
used  to  reference  this  control  state  at  any  time  in  the  future. 

The  first,  and  simplest,  way  this  can  be  accomplished  is  to  “package 
up"  the  entire  stack  by  copying  it  into  the  heap.  The  stack  can  then  be 
emptied  and  a  single  entry  pushed  onto  it.  This  entry  says,  in  essence,  that 
the  object  just  created  must  become  the  stack  if  control  ever  returns  to  this 
point  and  thus  the  continuation  will  be  copied  back  from  the  heap  into 
the  stack.  This  describes  the  original  implementation  of  continuations  in 
MIT  Scheme.  Measurements  have  shown  that  the  stack  rarely  grows  very 
deep  (over  100  entries)  even  in  lengthy  (syntactically)  recursive  programs, 
although  it  is  quite  easy  to  write  a  program  that  allows  the  stack  to  grow 
arbitrarily  deep.  Thus,  the  cost  of  copying  the  entire  stack  is  not  large,  and 
.since  call-with-current-contiimation  is  rarely  used,  this  implementa¬ 
tion  causes  no  major  problems.  Thus,  the  third  major  departure  from  the 
Algol  family  of  languages  is  the  ability  to  capture  the  stack  as  an  object  in 
the  programming  langtiage.  This  is  shown  as  the  third  major  difference  in 
Figure  A. 9. 

There  have  been  two  changes  to  the  implementation  of  call-with- 
current-continuation,  however,  as  a  result  of  the  work  on  MultiScheme. 
.4s  Will  be  discussed  in  Chapter  4,  this  primitive  is  used  to  support  task 
switching  in  MultiScheme  and  its  performance  is  therefore  much  more  im- 
jiortant  than  in  MIT  Scheme.  The  first  change  is  an  alternative  implemen¬ 
tation  of  the  interpreter  (and  compiler  interface)  allowing  the  stack  to  be 
allocated  in  the  heap  and  tiius  eliminates  the  need  to  copy  the  stack  at  the 
time  the  continuation  object  is  made.  Instead,  it  is  copied  incrementally 
whf’ii  control  returns  to  the  part  of  the  stack  that  existed  at  the  time  of 
the  call  to  call-with-current-continuation.  The  .second  change  was 
the  recognition  (as  explained  in  Chapter  4)  that  the  continuation  objects 
vised  for  task  switch  are  in  fact  never  rcu.sed.  Although  they  have  indefi¬ 
nite  ex'ent.  tht'V  are  created  and  u.sed  exactly  once.  .4  far  more  efficient 
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implementation  can  be  made  under  these  circumstances,  and  an  additional 
primitive,  non-reentrant-call-with-current-continuation  Wcis  added 
precisely  to  support  this  operation. 

A. 5  MIT  Scheme  as  a  System 

One  traditional  method  for  building  a  programming  system  is  to  divide  the 
task  into  three  parts:  language  implementation,  portable  language-beised 
library,  and  a  system-specific  library  for  interfacing  to  the  underlying  oper¬ 
ating  system  and  external  (non-language  specific)  facilities.  The  goal  of  this 
structure  is  to  facilitate  the  construction  of  portable  programs  (by  promot¬ 
ing  the  use  of  the  portable  library)  while  at  the  same  time  allowing  efficient 
access  to  non-portable  features  available  within  a  specific  implementation. 
Lisps  exhibit  a  similar  structure,  although  the  delineation  of  the  compo¬ 
nents  is  frequently  far  less  cleax  and  the  portable  library  tends  to  contain 
a  wider  range  of  procedures  than  those  of  many  other  systems. 

Since  standard  computer  architectures  do  not  provide  support  for  the 
garbage  collection  essential  to  the  Lisp  language,  most  Lisp  implementa¬ 
tions  are  forced  to  supply  their  own  memory  management  routines.  But 
once  the  Lisp  implementation  becomes  responsible  for  its  own  memory  man¬ 
agement  it  becomes  considerably  simpler  to  support  the  dynamic  allocation 
of  what  are  frequently  supplied  only  eis  static  objects  (tasks,  files,  and  so 
forth).  Thus  Lisp  systems  tend  to  build  very  large  portable  libraries  consist¬ 
ing  of  what  would  ordinarily  be  considered  “systems  code.”  Furthermore, 
there  is  a  bias  within  the  Lisp  community  to  provide  users  of  the  system 
with  access  to  a  very  wide  range  of  implementation  decisions  within  this 
kind  of  code.  Where  possible  the  system  builders  provide  “hooks”  for  users 
to  specify  their  own  extensions  to  a  framework  provided  and  maintained 
along  with  the  rest  of  the  Lisp  system. 

Lisp  systems  tend  to  be  built  in  four  parts.  There  is  a  core  language 
imph'mented  in  some  non-Lisp  language  or  the  result  of  cross-compiling 
carefully  written  Lisp  code.  An  extensible  set  of  primitive  procedures  pro¬ 
vides  the  very  low-level  support  needed  to  implement  the  rest  of  the  system 
is  generally  written  in  assembly  language  for  the  target  machine.  On  top 
of  these  a  set  of  system  procedures  (written  in  Lisp)  provides  an  interface 
between  the  external  environment,  the  core  language,  and  user  programs. 
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Finally,  a  system  supplied  library  of  general  utility  procedures  is  written  5 

using  the  other  three  parts  as  a  base.  A  large  component  of  this  report  is 
devoted  to  the  design  and  implementation  of  the  interface  procedures  for  a 
parallel  processor  system.  Since  these  procedures  are  closely  tied  to  the  un¬ 
derlying  language  core,  the  broad  outlines  of  this  core  must  be  understood 


before  their  implementation  can  be  explained. 

A. 5.1  Machine  Model  or  Core  Interpreter 

The  MIT  Scheme  system  is  built  to  run  on  a  virtual  machine  whose  origins 
are  in  the  Scheme  ’79[33]  and  Scheme  ’81[10]  projects.  Since  neither  of 
these  projects  (nor  their  successor  hardware  projects)  has  become  widely 
available,  a  portable  interpreter  has  been  implemented  to  simulate  the  in¬ 
struction  set  of  these  machines.  This  interpreter  is  the  basis  for  the  actual 
implementation  of  MultiScheme^. 

These  machines  are  based  around  the  ability  to  manipulate  typed  ob¬ 
jects.  In  the  current  implementation  all  objects  consist  of  two  parts:  the 
data  type  field,  and  the  value  field^.  The  value  field  can  contain  either  im¬ 
mediate  data  (for  example,  a  short  signed  integer)  or  more  commonly  an 
address.  The  machine  itself  runs  a  tree- structured  instruction  set  (known 
as  SCode).  SCode  is  little  more  than  a  parse  tree  for  the  program,  using  an 
opcode  for  each  of  the  Scheme  specieJ  forms  (plus  some  additional  ones  for 
combinations,  variables,  and  constants).  These  operations  are  encoded  in 
the  data  type  field.  The  machine  maintains  a  stack  of  continuations  as  de¬ 
scribed  in  Section  A. 4,  and  has  a  small  set  of  dedicated  registers  referenced 
implicitly  by  many  of  the  instructions. 

In  addition  to  this  simple  execution  engine,  there  is  a  linear  memory 
space  with  a  free  pointer  used  to  allocate  the  heap.  When  this  pointer 
is  incremented  beyond  some  specified  address,  the  processor  indicates  the 
(•(uidition  by  setting  one  of  a  number  of  interrupt  request  bits,  indicating 
that  a  garbage  collection  cycle  should  begin  as  soon  as  convenient.  These 

■’An  MIT  Scheme  compiler  is  being  currently  under  development.  This  compiler  accepts 
JUS  input  programs  written  in  the  instruction  set  of  these  machines,  performs  a  number  of 
optimizations,  and  generates  instructions  for  a  standard  computer  architecture. 

■'f'urrent ly,  the  data  type  field  is  7  bits  wide,  the  value  is  24  bits  wide,  and  there  is 
a  1  l)it  wide  field  called  the  "danger  bit  ”  This  danger  bit  is  being  phased  out  of  the 
im[dementatioti  atid  is  not  discussed  further  in  this  report. 
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bits,  along  \\ith  an  interrupt  request  mask  (set  under  program  control),  are 
sampled  periodically  and  cause  specific  Scheme  procedures  to  be  invoked. 
In  addition  to  this  garbage  collection  interrupt  a  number  of  other  interrupt 
conditions  are  treated  in  the  same  way.  A  priority  system  is  used  to  handle 
stacking  of  these  interrupt  conditions. 

Finally,  some  instructions  can  detect  error  conditions.  For  example, 
the  variable  reference  instruction  can  fml  if  a  variable  is  unbound  in  the 
current  environment.  In  these  cases  a  trap  into  Scheme  code  is  also  invoked. 
Since  these  errors  are  synchronous  with  instruction  execution  there  is  no 
need  for  the  interrupt  bits  or  priority  mechanism.  Any  instruction  that  can 
generate  an  error  is  required  to  “back  out”  and  leave  the  system  in  the  same 
state  it  was  in  prior  to  the  attempt  to  execute  the  instruction.  This  allows 
the  instruction  to  be  retried  later  if  the  error  condition  is  removed.  Thus 
executing  an  erroneous  instruction  appears  to  the  Scheme  system  as  though 
the  user  had  inserted  a  call  to  the  appropriate  error  handler  immediately 
prior  to  the  erroneous  instruction. 

In  order  to  allow  the  hardwaire  to  call  Scheme  procedures  in  case  of  errors 
or  interrupts,  the  machine  and  the  Scheme  runtime  system  communicate 
through  a  region  of  memory  known  to  both.  This  area  contains  the  Scheme 
procedure  object  to  invoke  for  each  interrupt  or  error  condition,  as  well  as 
other  information  accessible  to  both. 

A. 5. 2  Primitive  Procedures 

The  majority  of  the  core  interpreter  actually  consists  of  the  implemen¬ 
tation  of  a  (unfortunately)  large  number  of  primitive  procedures.  These 
procedures  can  be  roughly  divided  into  three  classes. 

1.  Unimplementable.  These  primitives  support  language  features  and 
cannot  be  coded  directly  in  Scheme  without  resorting  to  directly  read¬ 
ing  and  writing  absolute  memory  locations.  Sample  primitives  in  this 
category  are  garbage-collect,  general-car-cdr,  apply,  and  call- 
with-current-continuation.  The  majority  of  these  operations  are 
primitive  space  allocation  or  data  structure  referencing  operations. 

2.  Speed-up.  Most  of  the  primitives  are  supplied  only  to  make  the  opera¬ 
tions  faster  than  they  would  be  if  implemented  in  Scheme.  String  and 
list  search  operations,  case  conversion,  integer  restricted  arithmetic. 
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etc.  are  in  this  category.  The  theory  is  that  when  a  “good  enough 
compiler"  is  available,  these  will  be  supplied  a^  part  of  the  runtime 
system  (i.e.  written  in  Scheme)  rather  than  as  primitives. 

3.  External  interface.  About  one  fourth  of  the  primitives  in  the  MIT 
system  are  used  to  interface  with  objects  outside  of  the  Scheme  world. 
This  includes  file  operations,  date  and  time  handling,  and  so  forth. 
Most  of  these  primitives  are  not  needed  by  the  Scheme  system  directly, 
although  users  of  the  system  might  be  disappointed  if  there  were  no 
way,  for  example,  to  read  or  write  files. 

From  the  point  of  view  of  the  core  interpreter,  the  number  and  compo¬ 
sition  of  the  primitives  is  completely  immaterial.  They  have  a  standard 
interface  allowing  the  interpreter  to  check  that  the  correct  number  of  argu¬ 
ments  have  been  supplied  and  then  turn  control  over  to  the  primitive  itself. 
The  primitives  themselves  are  responsible  for  type  and  range  checking  of 
any  arguments,  and  they  use  a  well  defined  interface  to  signal  errors  or 
interrupt  conditions.  Some  primitives  also  alter  the  flow  of  control  of  the 
interpreter  itself  (for  example,  the  apply  primitive  stores  information  on 
the  continuation  stack  and  then  requests  that  the  interpreter  proceed  on 
to  a  procedure  application  rather  than  by  processing  the  returned  value  of 
the  primitive). 

A. 5. 3  System  Code 

The  third  and  fourth  components  of  the  MIT  Scheme  system  constitute 
the  “runtime  system,"  written  in  Scheme  itself.  As  might  be  expected,  it 
really  consists  of  two  categories  of  code.  There  is  a  “low  level"  portion  of 
the  system  dealing  with  the  details  of  interfacing  with  the  interpreter  core, 
and  an  extensive  set  of  utility  procedures  designed  for  users  of  the  system. 

The  runtime  system  (low  level  portion)  is  itself  organized  as  a  number 
of  interrelated  packages  (i.e.  lexical  environments)  of  procedures.  The  fol¬ 
lowing  list  is  incomplete,  but  describes  a  number  of  these  packages  with 
particular  reference  to  parts  that  are  impacted  by  the  system  changes  de¬ 
scribed  in  this  report. 

•  System  initialization,  binary  file  loading  and  dumping. 
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•  The  reader  and  printer  handle  input  and  output  operations  and 
interaction  with  the  host  operating  system. 

•  The  parser  and  unparser  convert  between  the  external  (character 
string)  and  internal  (typed  pointer)  representations  of  objects.  User 
operations  such  as  read  depend  on  the  reader  to  gather  characters 
from  the  input  source,  then  pass  them  to  the  parser  for  conversion 
to  the  internal  object  representation. 

•  The  syntaxer  and  unsyntaxer  convert  between  the  program  (SCode) 
and  surface  syntax  (list)  representations  of  programs. 

•  Type  dispatching  and  SCode  abstraction  allow  a  convenient  interface 
to  the  underlying  representation  of  objects  and  programs. 

•  The  REP-Loop  package  allows  the  construction  and  stacking  of  user 
interaction  procedures.  It  is  tightly  coupled  with  the  error  and  inter¬ 
rupt  systems  (below).  The  interaction  of  this  structure  with  exception 
handling  is  discussed  in  Appendix  D. 

•  The  interrupt  and  error  system  respond  to  traps  from  the  core 
interpreter.  Changes  and  extensions  to  the  interrupt  system  (in  par¬ 
ticular  to  support  the  initiation  of  garbage  collection)  are  discussed 
in  Section  3.1.1.  The  error  system  reacts  to  most  errors  by  using 
the  REP-Loop  component  of  the  system  code.  Exception  handling  is 
not  a  part  of  the  error  system,  but  is  closely  related  to  the  REP-Loop 
component  as  well. 

•  The  scheduler  is  a  component  of  the  system  added  in  the  conver¬ 
sion  from  MIT  Scheme  to  MultiScheme.  Chapter  4  discus.ses  this 
component  in  detail. 


A. 6  Summary 

Th('  Scheme  language  as  u.sed  in  this  report  has  a  very  simple  syntax  (Sec- 
don  A.l.  Figure  A.l.  and  Figure  A. 4).  In  addition  to  the  ordinarily  encoun¬ 
tered  procedures  of  CommonLisj),  some  of  the  examples  in  this  report  use 
the  ])roee(lures  described  in  Figure  A. 2.  The  language  makes  heav'y  use  of 
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the  ability  to  create  and  return  procedures  on  the  fly  (in  older  Lisp  parlance 
it  "solves  the  upward  and  downward  funarg  problems”),  and  examples  of 
this  ability  are  provided  in  Figures  A. 5  and  A. 6. 

The  Scheme  dialect,  but  not  CommonLisp,  provides  the  ability  to  con¬ 
vert  the  otherwise  implicit  continuation  for  an  expression  into  an  object  of 
the  language.  This  allows  control  structures  of  arbitrary  complexity  to  be 
built  within  the  language.  Figure  A. 8  shows  one  very  simple  use  of  this 
ability.  This  usage  is  the  key  to  understanding  the  task  switch  of  Multi- 
Scheme  (discussed  in  Chapter  4).  The  implementation  of  continuations  in 
the  MIT  Scheme  system  is  based  on  a  stack  model  and  the  key  ideas  of 
reductions  and  subproblerns  as  presented  in  Section  A. 4. 3. 

Finally,  the  MIT  Scheme  implementation  is  layered  in  four  constituent 
parts.  There  is  a  core  interpreter  (Section  A. 5.1),  an  extensible  set  of 
primitive  procedures  (Section  A. 5.2),  system  code  (Section  A. 5. 3)  and  a 
library  of  utility  procedures.  Errors  and  interrupts  are  reflected  into  the 
system  code  by  calling  Scheme  procedures  provided  as  part  of  the  standard 
system.  A  set  of  hooks  allows  users  to  customize  the  Scheme  runtime 
system  by  supplying  their  own  code  for  use  in  a  variety  of  well  defined 
circumstances. 


Appendix  B 

Implementation  of  the  Pipeline 


This  appendix  includes  the  code  used  to  implement  the  pipeline  example 
of  Section  5.3.  This  code  is  presented  complete  and,  with  the  exception  of 
additional  comments,  unedited. 

B.l  Locks  for  Serializing  Access 

Recall  that  (set-car-if-eq? !  a  b  c)  is  an  atomic  operation  that  either 
operates  like  (set-car!  a  b)  (if  (car  a)  is  currently  c),  or  it  does  noth¬ 
ing.  It  returns  a  if  the  modification  takes  place  and  ’  ()  if  not. 

(define  (lock  obj)  ;  Internal 

(if  (null?  (set-car-if-eq?!  obj  ’LOCKED  ’UNLOCKED)) 

(lock  obj))) 

(define  (unlock  obj)  (set-car!  obj  ’UNLOCKED))  ;  Internal 

(define  (make-lock  (make-lock  value)  (cons  ’UNLOCKED  value))) 

(define  lock. value  edr) 

(define  set-lock. value!  set-edr!) 

(define  (atomically  object  procedure) 

(lock  object) 

(let  ((result  (procedure))) 

(unlock  object) 
result) ) 
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B.2  General  Utilities 

(define  (make-placeholder) 

((access  make-future  scheduler) 

’MAKE-PLACEHOLDER  ’MAKE-PLACEHOLDER  "Waiting  Forever")) 

(define  (make-list  count  object-generator) 

(let  loop  ((answer  ’()) 

(count  count)) 

(if  (=  count  0) 
answer 

(loop  (cons  (object-generator)  answer) 

(-1+  count))))) 

(define  (make-vector  count  obj-gen) 

(list->vector  (make-list  count  obj-gen))) 

(define  (start -pipeline  pipeline  vector) 

(if  (and  (pair?  pipeline) 

(eq?  (car  pipeline)  ’INPUT-NODE)) 

(vector  ’MESSAGES  (cdr  pipeline)) 

(error  "START-PIPELINE:  not  an  input  node"  pipeline))) 

Refer  to  procedure  make-pipe-vector  in  Section  B.6  to  understand 
how  pipeline  vectors  handle  the  messages  operation. 

B.3  Hash  Tables 

One  and  two-dimensional  tables  are  provided,  based  on  hash  numbers  gen¬ 
erated  using  the  interning  service  described  in  Section  2.2.2.  As  mentioned 
there,  this  implementation  should  really  use  weak  cons  cells,  but  I  was  lazy. 
It  is  important  that  the  tables  not  retain  the  objects  entered  in  the  table, 
since  there  is  no  code  to  remove  entries  when  they  are  no  longer  needed. 
That  should  have  been  implemented,  too. 
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(define  (meike-table)  (make-lock  ’())) 

(define  (hash-add  table  hash-code  adder) 

(let  ((first-try  (assq  hash-code  (lock. value  table)))) 
(if  first-try 

(cadr  first-try) 

(atomically  table 
(lambda  () 

(let  ((second-try  (assq  hash-code 

(lock. value  table)))) 

(if  second-try 

(cadr  second-try) 

(let  ((result  (adder))) 

(set-lock. value!  table 
(cons  (list  hash-code  result) 

(lock. value  table))) 
result)))))))) 

(define  (Id-add  table  entry  adder) 

(let  ((hash-code  (object -hash  entry))) 

(hash-add  table  hash-code  adder))) 


(define  (2d-add  table  first  second  adder) 

(let  ((first-hash  (object-hash  first)) 

(second-hash  (object-hash  second))) 

(let  ((sub-table  (hash-add  table  first-hash 

(lambda  ()  (make-table))))) 
(hash-add  sub-table  second-hash  adder)))) 
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B.4  Simple  Pipeline  Constructors 

The  pipeline  is  simply  a  list  of  messages  to  be  processed  by  the  objects 
flowing  through  the  pipe.  During  construction,  a  “dangling  end”  has  a 
message  in  the  car,  and  a  ’  ()  in  the  cdr. 

;  Internal  procedures : 

;  (1)  Add  a  new  operation  to  the  end  of  am  existing  pipe 
(define  (next-op!  pipe  op) 

(let  ((result  (list  op))) 

(set-cdr!  pipe  result) 
result) ) 

;  (2)  Join  two  existing  pipes  so  the  first  flows  into  the  second 
(define  (weld!  in-pipe  out-pipe) 

(set-cdr!  in-pipe  out-pipe)) 

;  External  procedures,  see  Section  5.3.1; 

(define  (extend  pipe  operation)  (next-op!  pipe  operation)) 

(define  (make-input)  (list  ’INPUT-NODE)) 


B.5  Complicated  Pipeline  Constructors 

The  main  pipeline  constructors  operate  by  adding  a  procedural  message  to 
the  growing  pipeline.  In  the  caise  of  fork,  the  message  causes  an  incoming 
object  to  make  new  objects  with  copies  of  its  current  state,  each  of  which 
propagates  down  a  different  output  branch: 

(define  (fork  n  input)  ;  Yields  list  of  n  output  pipes 

(let  ((result  (make-list  (-1+  n)  (lambda  ()  (list  ’FORK-START))))) 
(define  (fork-operation  obj) 

(let  ((obj-num  (obj  ’OBJECT-NUMBER)) 

(vector  (obj  ’VECTOR))) 

(map  (lambda  (pipe) 

(let  ((new-object  (obj  ’NEW-OBJECT  (obj  ’COPY-STATE)))) 
(new-object  vector  obj-num  (cdr  pipe)))) 
result) ) ) 

(cons  (next-op!  input  fork-operation)  result))) 
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The  remaining  constructors  all  require  synchronization  of  arriving  ob¬ 
jects.  They  use  tables  (constructed  from  the  operators  above)  to  locate 
corresponding  objects  from  the  Scune  vector  eis  they  pass  through  a  partic¬ 
ular  point  in  the  pipeline. 

At  a  join  point,  the  first  object  corresponding  to  a  vector  and  offset 
within  that  vector  proceeds  on  to  the  output  pipeline  after  performing  the 
user-specified  join  operation.  Objects  arriving  from  other  input  pipeline 
branches  for  the  same  vector  and  offset  are  forced  to  halt  after  they  make 
their  state  available  to  the  first  object: 

(define  (join  combiner  inputs) 

(let  ((count  (length  inputs)) 

(result  (list  ’IDLE)) 

(my-table  (make-table))) 

(define  (make-operation  input -number) 

(lambda  (obj) 

(let  ((vector  (obj  ’VECTOR)) 

(object -number  (obj  ’OBJECT-NUMBER)) 

(state  (obj  ’GET-STATE)) 

(first-one-through?  iF)) 

(let  ((mates  (2d-add  my-table  vector  object-number 
(lambda  () 

(set!  first-one-through?  iT) 

(make-vector  count  make-placeholder))))) 
(determine!  (vector-ref  mates  input-number)  state) 

(if  first-one-through? 

(obj  combiner  (vector->list  mates)) 

(obj  ’HALT)))))) 

(define  (loop  n  inputs) 

(if  (null?  inputs) 
result 
(begin 

(weld!  (next-op!  (car  inputs)  (ir  -operation  n)) 
result) 

(loop  (1+  n)  (cdr  inputs))))) 

(loop  0  inputs))) 

At  output  and  interact  points,  each  object  looks  to  see  if  any  other 
ol)jects  from  the  same  vector  have  arrived.  If  so,  it  merely  stores  its  state 
away  in  the  output  structure  already  created,  then  halts.  If  not  it  creates 
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a  data  structure  for  objects  that  arrive  later: 

(define  (add-output  pipe)  ;  Yields  list  of  answers 

(let  ((my-table  (make-table)) 

(result  (make-placeholder))) 

(define  (output-operation  obj) 

(let  ((vector  (obj  ’VECTOR)) 

(object-number  (obj  ’OBJECT-NUMBER)) 

(my-state  (obj  ’GET-STATE))) 

(let  ((result-vector 

(Id-add  my-table  vector 
(lambda  () 

(let  ((next-out  (make-placeholder)) 

(this-out  (make-list  (vector  ’SIZE) 

make-placeholder) ) ) 
(determine!  result  (cons  this-out  next-out)) 
(set!  result  next-out) 

(list->vector  this-out)))))) 

(determine!  (vector-ref  result-vector  object-number) 
my-state)))) 

(next-op!  (next-op!  pipe  output-operation)  ’HALT) 
result) ) 

(define  (add-interactor  pipe  interactor) 

(let  ((my-table  (maike-table)) 

(result)) 

(define  (interactor-operation  obj) 

(let  ((obj-vector  (obj  ’VECTOR))) 

(let  ((data  (Id-add  my-table  obj-vector 
(lambda  () 

(let  ((1  (make-list  (obj-vector  ’SIZE) 

make-placeholder) ) ) 
(cons  (list->vector  1)  1)))))) 

(determine ! 

(vector-ref  (car  data)  (obj  ’OBJECT-NUMBER)) 

(obj  ’GET-STATE)) 

(obj  interactor  (cdr  data))))) 

(set!  result  (extend  pipe  interactor-operation)) 
result) ) 
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B.6  Creating  a  Pipe  Vector 

Pipeline  vectors  are  message  receiving  objects.  They  take  either  a  messages 
message,  used  to  associate  the  vector  with  a  given  pipeline  input  port  (see 
start-pipeline,  in  the  general  utilities  section),  or  a  size  message  to  re¬ 
port  the  number  of  objects  in  the  vector.  All  of  the  objects  in  the  vector 
are  created  using  make-object,  and  they  are  activated  by  specifying  the 
vector  to  which  they  belong,  the  offset  within  that  vector,  and  the  messages 
(pipeline)  they  are  to  process: 

(define  (make-pipe-vector  elements) 

(let  ((message-list  (make-placeholder)) 

(size  (length  elements))) 

(define  (the-vector  message  .  additional) 

(cond  ((eq?  message  ’MESSAGES) 

(determine!  message-list  (car  additional))) 

((eq?  message  ’SIZE)  size) 

(else  (error  "Bad  message  to  vector"  message)))) 

(define  (loop  number  elems) 

(if  (null?  elems) 

the-vector  ;  Value  returned:  the  vector 

(begin  ;  Activate  the  objects 

((car  elems)  the-vector  number  message-list) 

(loop  (1+  number)  (edr  elems))))) 

(loop  0  elements))) 


B.7  Task  Creation  and  Removal 

(define  *all-objects*  (make-lock  (list  ’OBJECTS))) 

(define  (make-cell  before  me  after) 

(cons  (cons  before  me)  after)) 

(define  (before  cell) 

(if  (or  (future?  cell) 

(future?  (car  cell))) 

(error  "Before  trouble"  cell)) 

(caar  cell)) 
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(define  (after  cell) 

(if  (future?  cell) 

(error  "After  trouble"  cell)) 

(cdr  cell)) 

(define  (set-before!  cell  value) 

(if  (or  (future?  cell) 

(future?  (car  cell))) 

(error  "Set-Before!  trouble"  call)) 

(set-car!  (car  call)  value) 

’neu-bef ore) 

(define  (set-after!  cell  value) 

(if  (future?  cell) 

(error  "Set-After!  trouble"  cell)) 

(set-cdr!  call  value) 

’new-after) 

(define  (kill-task  call) 

(atomically  •all-objects* 

(lambda  () 

(let  ((bafore-me  (before  cell)) 

(after-me  (after  cell))) 

(set-after!  before-me  after-me) 

(if  after-me 

(set-before!  after-me  before-me))))) 

((access  next  scheduler))) 

(define  (record-task!  me) 

(atomically  •all-objects* 

(lambda  () 

(let  ((after-me  (after  (lock. value  •all-objects*))) 
(before-me  (lock. value  •all-objects*))) 

(let  ((my-cell  (make-cell  before-me  me  after-me))) 
(if  after-me  (set-before!  after-me  my-cell)) 
(set-after!  before-me  my-cell) 
my-cell))))) 
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B.8  Making  an  Object 

A  simplified  version  of  this  procedure  was  shown  in  Figure  5.11  on  page  152. 
The  complete  code  for  handling  all  operations  is: 

(define  (make-object  state  copy-state  user-code) 

(let  ((my-cell) 

(vector) 

(obj  ect-number) 

(the-messages  (make-placeholder) ) ) 

(define  (loop  message-list) 

(define  (standard-handler  m  #! optional  arg) 

(cond 

((procedure?  m)  (m  steuidard-handler) ) 

((eq?  m  ’halt)  (kill-task  my-cell)) 

((eq?  m  ’idle)  ’IDLED) 

((eq?  m  ’new-object)  (make-object  arg  copy-state  user-code)) 
((eq?  m  ’obj ect-number)  obj ect-number) 

((eq?  m  ’vector)  vector) 

((eq?  m  ’get-state)  state) 

((eq?  m  ’set-state!)  (set!  state  arg)) 

((eq?  m  ’copy-state)  (copy-state  state)) 

(else  (set!  state 

(user-code  state  m  (if  (unassigned?  arg)  ’()  arg)))))) 
(standard-handler  (car  message-list)) 

(loop  (cdr  message-list))) 

(set!  my-cell  (record-task!  (future  (loop  the-messages)))) 

(lambda  (the-vector  number  messages) 

(set!  vector  the-vector) 

(set!  object-number  number) 

(determine!  the-messages  messages)))) 
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Appendix  C 

Performance  Measurements 


The  MultiScheme  project  was  not  primarily  concerned  with  issues  of  imple¬ 
mentation  efficiency,  but  these  are  clearly  important  to  any  serious  parallel 
programming  system.  This  appendix  provides  measurements  of  some  of  the 
important  operations  internal  to  the  MultiScheme  interpreter,  primarily  to 
provide  a  baseline  for  comparison  with  future  versions  of  the  system. 

Performance  measurement  is  an  area  which  by  rights  requires  careful 
planning  and  a  well  chosen  goal  —  typically  aimed  at  discovering  the  im¬ 
portance  of  one  aspect  of  the  implementation  to  the  overall  performance 
of  a  large  system.  These  measurements  are  not  part  of  such  an  effort,  and 
their  utility  is  correspondingly  limited. 

C.l  The  Measured  System 

The  measurements  are  taken  from  a  simulator  for  MultiScheme  that  runs 
on  the  Hewlett-Packard  Series  9000  Model  320  computer,  based  on  a  Mo¬ 
torola  MC68020  processor  running  at  16.67  MHz  using  a  16-bit  memory  bus 
with  a  large  (roughly  lOOK  line)  cache  between  the  processor  and  memory. 
The  simulator  is  written  in  C,  sharing  almost  all  of  the  code  with  the  ac¬ 
tual  implementation  of  MultiScheme  on  the  BBN  Butterfly.  The  primary 
differences  between  the  two  implementations  are: 

•  The  Butterfly  is  a  true  multiprocessor,  while  the  HP  machine  is  a  stan¬ 
dard  sequential  machine.  The  simulator  can  be  run  with  or  without 
timer-driven  scheduling  interrupts.  For  these  measurement  no  timer 
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was  used  since  the  examples  were  either  sequential  or  were  deliber¬ 
ately  intended  to  measure  the  overhead  of  task  switches  at  controlled 
times. 

•  The  Butterfly  hardware  is  b^lsed  on  an  8  MHz  processor  (normally 
a  Motorola  6S000  but  a  Motorola  G8020  is  avmlable  as  an  option) 
and  a  microcoded  co-processor  to  provide  a  virtual  memory  system. 
The  co-processor  is  responsible  for  all  transactions  across  the  But¬ 
terfly  Switch.  References  to  memory  that  is  physically  located  with 
the  processor  making  the  reference  are  faster  by  a  constant  factor 
(which  depends  on  the  configuration  of  the  particular  Butterfly)  than 
are  references  to  any  other  memory.  Switch  transactions  take  place 
in  16-bit  quantities,  but  the  co-processor  can  be  used  to  implement 
atomic  32-bit  transfers  at  the  cost  of  some  set-up  time.  Since  the 
MultiScheme  interpreter  uses  a  combination  of  both  atomic  zind  non- 
atomic  transfers,  memory  reference  times  are  not  comparable  on  the 
two  implementations.  Typical  memory  access  times  are  2  microsec¬ 
onds  for  local  memory  and  6  microseconds  for  remote  memory  [1]. 

•  The  Butterfly  memory  management  is  based  on  the  garbage  collec¬ 
tor  described  by  Courtemanche[14].  It  pre-allocates  portions  of  the 
address  space  to  each  processor,  and  a  garbage  collection  is  initi¬ 
ated  when  any  processor’s  area  is  filled.  This  alters  the  frequency  of 
garbage  collection  on  the  Butterfly  by  comparison  to  the  HP.  None 
of  the  performance  meeisurements  include  garbage  collection  activity 
on  the  HP. 

•  The  queue  of  tasks  awaiting  processors  is  provided  using  the  Butter¬ 
fly’s  atomic  “dual  queue”  operations[2]  operations.  These  are  simu¬ 
lated  using  primitive  procedures  (written  in  C)  for  queue  manipula¬ 
tion  on  the  HP.  Similarly,  the  primitive  procedures  global-inter¬ 
rupt  and  await -synchrony  are  simulated  on  the  HP. 

•  The  operating  system  on  the  HP  machine  is  HP-UX^^,  a  version 
of  Unix"^^h  The  Butterfly  runs  the  ChrysalisT^[2]  operating  system, 
developed  for  high-pe-formance  data  communications  applications  on 
a  multi-processor. 
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The  effects  of  these  differences  are  quite  difficult  to  quantify.  The  author 
therefore  suggests  that  these  numbers  not  be  considered  to  have  any  validity 
with  respect  to  an  actual  parallel  processor  implementation.  In  fact,  I 
anxiously  await  the  release  of  the  BBN  Butterfly  Lisp  system  so  that  a 
realistic  set  of  measurements  can  be  undertaken. 

One  further  caution:  the  system  being  measured  is  fully  interpreted. 
The  MultiScheme  system  code  used  during  the  measurements  contains  a 
scheduler  that  corresponds  to  the  scheduler  shown  in  Chapter  4,  but  with 
three  significant  differences: 


1.  The  scheduler  actually  measured  does  not  permit  the  independent 
creation  of  tasks  and  placeholders.  In  fact,  the  two  data  structures 
described  in  Chapter  4  are  combined  in  one. 

2.  The  race  conditions  inherent  in  the  implementations  of  Chapter  4  are 
not  present  in  the  measured  version.  The  scheduler  measured  runs 
unmodified  on  the  Butterfly,  and  no  races  have  been  detected. 

3.  The  measured  version  of  the  scheduler  uses  a  primitive  (written  in  C) 
to  implement  a  faster  version  of  call-with-current-continuation 
for  use  solely  in  task  switching.  This  new  primitive  does  not  permit 
the  continuation  it  creates  to  be  used  more  than  once,  and  is  known 
(informally)  as  task-catch.  Additionally,  part  of  the  code  for  spawn- 
task  (whose  implementation  was  not  shown  in  Chapter  4)  is  written 
ill  Scheme,  and  part  is  written  in  C. 


C.2  Measurement  Technique 

For  each  item  measured,  a  small  section  of  code  was  written  that  exercised 
the  desir(’d  operation.  Because  the  clock  resolution  available  was  1  tick 
every  0.01  seconds,  the  code  was  repeated  50  times  in  sequence  and  the 
total  time  for  the  50  repetitions  was  merisured.  This  group  of  50  operations 
wtis  then  repeated  20  times  (for  a  total  of  1000  operations),  separated  by  a 
garbage  collection. 

L  sing  this  technique,  calls  to  the  procedure  x  defined  by 

(define  (x)  3) 
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were  measured  for  purposes  of  normalization.  A  single  call  to  X,  measured 
in  this  way,  took  0.360  milliseconds  (ms).  For  comparison  procedure  calls 
of  1,  2,  6,  and  12  arguments  were  measured.  These  took  0.396,  0.396^ 
0.576  and  0.781  milliseconds,  respectively. 

Because  of  concern  over  the  accuracy  of  the  clock,  the  remaining  num¬ 
bers  were  normalized  with  respect  to  the  time  required  to  perform  a  call 
to  this  procedure,  x.  This  time  unit,  0.36  milliseconds,  is  referred  to  as  a 
"tick".  The  table  in  Figure  C.l  summarizes  the  results. 

C,3  Analysis:  The  Cost  of  a  future 

In  order  to  better  understand  the  cost  of  touching  a  placeholder  that  does 
not  yet  have  a  value  (131.6  ticks  from  Figure  C.l),  a  number  of  additional 
measurements  were  made.  The  cost  of  the  procedure  that  is  used  within 
the  scheduler  to  handle  this  case  was  measured  (this  procedure  is  shown 
evs  await-placeholder  in  Figure  4.10  on  page  111).  The  time  required  for 
this  procedure,  combined  with  a  cadi  to  task-catch,  was  48.65  ticks,  so  the 
cost  of  await-placeholder  itself  is  approximately  39.12  ticks.  In  addition, 
a  measurement  of  the  time  required  to  release  the  current  processor  and  get 
it  back  again  (roughly  the  cost  of  a  call  to  saving-state  and  then  using 
the  underlying  task  queues  to  save  and  restore  that  state)  was  39.8  ticks. 

Using  these  measurements,  we  can  compute  the  amount  of  time  we 
would  expect  a  (touch  (future  3))  to  take,  as  shown  in  Figure  C.2.  The 
difference  between  the  measured  cost  of  131.6  ticks  and  the  computed  value 
of  137.0  ticks  (from  Figure  C.2)  is  well  within  the  margin  of  error  in  these 
timing  measurements,  which  have  been  observed  to  fluctuate  by  as  much  as 
b%  (presumably  due  to  the  cost  of  certain  critical  Unix  background  jobs). 


'The  Sclieme  interpreter  is  optimized  for  one  and  two  argtiment  function  calls  (but  not 
zero  argutnents),  lending  credence  to  this  first  pair  of  numbers. 

I 

1 

i 

I 


\ 


C.3.  ANALYSIS:  THE  COST  OF  A  FUTURE 


231 


Code 

Time 

Notes 

( ticks  ) 

(ms.) 

(touch 

(future  3)) 

131.6 

47.4 

variable  reference 

0.7 

0.3 

For  a  local  variable 

(launbda  (x)  x) 

0.2 

0.1 

Creating  a  closure 

(future  3) 

19.0 

6.8 

Queues  new  task  and  returns  to 
spawning  tcisk 

determine ! 

13.0 

4.7 

touch 

1.2 

0.4 

Placeholder  that  has  a  value 

touch 

0.9 

0.3 

Non-placeholder  (eg-  af¬ 

ter  a  garbage  collection  replaces  a 
placeholder  with  its  value) 

task  dist. 

1.8 

0.7 

Store  and  retrieve  a  value  on  the 
task  distribution  queue 

enciueue 

5.3 

2.0 

Add  task  to  those  waiting  for 
placeholder’s  value 

dequeue 

5.1 

1.8 

Remove  task  from  those  waiting 
for  placeholder’s  value 

call/cc 

11.0 

4.0 

Make  a  continuation  using  call- 
with-current-continuation  and 
then  invoking  it 

task-catch 

9.5 

3.4 

Optimized  version  of  above,  used 
for  task  switch 

Note: 

As  mentioned  in  the  text,  all  timings  are  reported  in  ticks  corresponding  to  the  time 
required  to  call  a  procedure  of  no  arguments.  This  time  was  measured  as  0.36  milliseconds, 
and  includes  the  costs  of:  one  lexical  variable  look-up  (for  the  variable  X);  allocating  and 
initializing  an  environment  frame;  saving  the  current  continuation;  evaluating  a  constant; 
and  restoring  the  previous  continuation. 

Figure  C.l:  Internal  Timing  Measurements 
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Item 

Time 
( ticks  ) 

Description 

a 

19.0 

Creating  the  task  and  placeholder 

b 

39.1 

Touching  the  undetermined  placeholder  and  calling 
await -placeholder 

c 

39.8 

TEisk  switch  from  parent  to  child  task 

d 

13.0 

Determine!  value  of  placeholder 

e 

5.1 

Dequeue  parent  task  from  queue  waiting  for  place¬ 
holder’s  value 

f 

39.8 

Task  switch  from  child  back  to  parent  task 

g 

1.2 

Touch  of  the  placeholder  after  value  is  known 

157.0 

Sub-total 

-20.0 

Item  b  includes  a  suspend  of  the  parent  task,  as 
does  item  c.  Assuming  that  the  cost  of  suspension 
is  roughly  half  the  cost  of  a  full  task  switch,  this 
double  count  heis  a  cost  of  roughly  20 

137.0 

Total 

Note: 

See  the  text  and  Figure  C.l  for  an  explanation  of  the  time  unit. 

Figure  C.2:  Components  of  the  Cost  of  a  Fhture 


Appendix  D 

Exception  Handling  in 
MultiScheme 

Section  3.3  explained  how  fluid  variables  can  be  used  to  provide  storage  on 
a  per-task  basis.  In  passing,  it  was  mentioned  that  the  ability  to  use  first- 
class  continuations  to  exit  from  the  body  of  a  fluid-let  can  lead  to  com¬ 
plications,  and  the  implementation  mechanism  described  in  Section  3.3.4 
correctly  handles  these  difficulties.  This  implementation  was  newly  intro¬ 
duced  in  MultiScheme  to  replace  a  mechanism  based  on  the  notion  of  twin 
dynamic  state  and  control  state  introduced  in  MIT  Scheme  following  the 
work  of  Hanson  and  Lamping[28]. 

These  notions  are  useful  beyond  their  ability  to  implement  fluid  vari¬ 
ables;  they  provide  the  base  on  which  exception  handling  mechanisms  can 
be  built.  This  section  describes  these  twin  notions,  how  they  interact,  and 
how  they  could  be  used  in  MultiScheme  to  build  the  kind  of  exception 
handling  facilities  available  in  other  languages.  MultiScheme,  at  present, 
provides  only  the  basic  support  for  control  and  dynamic  state  spaces.  None 
of  the  applications  developed  so  far,  nor  the  system  itself,  have  had  need 
of  an  exception  handling  mechanism.  As  a  result,  this  section  describes  a 
basic  mechanism  and  several  ways  to  use  that  mechanism.  It  provides  no 
final  system  designer’s  choice  —  only  application  of  the  mechanism  to  real 
piroblems  will  provide  the  insight  needed  to  make  sirch  a  choice. 
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D.l  Twin  Notions 

The  addition  of  first-class  continuation  objects  to  Scheme  introduced  an 
unprecedented  amount  of  power  into  the  language.  Control  constructs  such 
as  setjmp  and  longjmp  of  many  C  implementations[4],  co- routines,  and 
even  multiprocessing  systems  can  be  conveniently  expressed.  The  control 
state  of  the  system  can  be  captured  at  any  point  in  time  and  reactivated 
at  any  other  time,  and  even  reactivated  multiple  times.  An  excellent  and 
provocative  use  of  this  last  ability  is  demonstrated  by  Rozas[51],  where  the 
ability  to  reactivate  a  control  state  is  used  to  simulate  the  fundamental 
operations  of  quantum  mechanics.  The  creation  and  manipulation  of  these 
control  state  objects,  called  continuations,  is  described  in  Section  D.3. 

But  this  kind  of  expressive  power  is  rarely  added  to  a  language  with¬ 
out  introducing  some  corresponding  difficulty.  In  the  case  of  Scheme  the 
difficulty  comes  from  the  kind  of  problems  that  were  mentioned  when  fluid 
variables  were  introduced  in  Section  3.3.3.  There  are  times  when  a  pro¬ 
gram  needs  to  set  up  some  state  that  must  be  maintained  for  the  duration 
of  a  particular  body  of  code.  In  the  example  of  Section  3.3.3  this  state 
consisted  of  the  storage  location  associated  with  the  variable  radix.  Other 
common  cases  include  opening  and  closing  files,  locking  and  unlocking  data 
structures,  and  saving  and  restoring  hardware  registers.  One  of  the  fastest 
growing  sets  of  functions  in  the  MIT  Scheme  system  are  the  “with-...” 
functions  (with-interrupt-mask,  with-output-to-f ile,  with-syntax- 
table,  etc.)  used  to  perform  this  sort  of  “modify. .  .execute. .  .unmodify” 
operation. 

In  all  of  these  cases  the  intention  is  to  perform  some  action  when  control 
exits  a  block  of  code,  independent  of  the  mechanism  used  to  exit  the  block. 
This  is  a  fairly  standard  problem,  often  handled  as  a  special  case  of  the 
more  general  problem  of  handling  exceptions  or  errors.  An  overview  of  two 
different  solutions  from  other  languages  is  supplied  in  Section  D.2. 

Because  Scheme  provides  the  ability  to  re-enter  a  region  of  execution, 
however,  a  number  of  new  difficulties  arise.  If  a  region  of  code  requires  a 
certain  state  to  be  in  existence  while  it  is  executing,  then  reentry  to  the 
region  must  reestablish  that  state.  In  order  to  support  this  requirement 
MIT  Scheme  provides  a  data  structure,  called  a  dynamic  state  space,  that 
maintains  the  procedures  required  for  transitions  into  and  out  of  execution 
regions.  This  mechanism  is  described  in  Section  D.4.  But  the  dynamic 
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and  control  states  must  be  related  to  one  another.  Transitions  within  the 
control  space  correspond  to  the  exiting  of  one  block  of  code  and  the  entry 
into  another.  The  dynamic  state  records  the  transition  functions  that  must 
therefore  be  invoked.  This  interconnection  is  described  in  Section  D.5. 


D.2  Exception  Handling 

Languages  like  Ada[44],  Clu[38]  and  Mesa[4l]  provide  syntactically  limited 
ways  of  exiting  a  block  of  code.  One  way  typically  corresponds  to  an 
ordinary  exit  and  is  implicit  when  the  body  of  code  completes  in  the  normal 
manner.  Another  syntactically  distinct  mechanism  is  used  if  the  body  of 
code  completes  in  an  unusual  manner.  These  unusual  exits  are  typically 
called  signals  or  exceptions. 

Users  can  provide  exception  handlers  for  certain  blocks  of  code.  When 
a  program  raises  an  exception  (either  by  using  the  appropriate  syntax  or 
implicitly  from  errors  detected  by  the  runtime  system)  the  block  of  code 
that  has  most  recently  been  entered,  has  not  yet  been  exited,  and  that 
contains  an  exception  handler  for  the  particular  exception  is  located  and 
the  handler  is  invoked.  A  block  of  code  that  exits  normally  does  not  activate 
any  of  the  exception  handlers  and  furthermore  these  exception  handlers  are 
removed  from  consideration  when  future  exceptions  are  raised  (the  block  is 
now  "exited” ).  If  the  code  for  an  exception  handler  is  required  to  run  for  all 
possible  ways  of  exiting  the  block  (as  in  the  example  of  Section  3.3.2)  the 
code  for  the  block  terminates  by  reusing  the  appropriate  exception  condition 
itself  instead  of  using  the  ordinary  return  mechanism. 

Details  of  the  mechanisms  differ,  but  in  general  these  systems  provide  for 
.several  important  cases.  Exception  handlers  may  be  si’pplied  for  a  specific 
exception,  a  class  of  exceptions,  and  “all  exceptions  that  aren’t  otherwise 
handled."  Raising  an  exception  can  pass  user  selected  data  to  the  exception 
handlers.  When  an  exception  handler  is  invoked  it  may  be  permitted  to  re- 
iiivoke  the  exception  after  performing  some  code,  thus  becoming  invisible 
except  for  providing  clean-up  code.  Or  it  may  raise  a  different  exception, 
or  force  the  normal  exit  from  its  own  block  rather  than  continuing  the 
propagation  of  an  exception.  Or  it  may  cause  the  procedure  that  raised  the 
exception  to  appear  to  return  a  selected  value. 

In  summary,  these  lang\iages  (a)  annotate  blocks  of  code  with  exception 
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haiullers;  (h)  distiiignish  normal  return  from  unusual  block  exit;  (c)  provide 
a  path  of  communication  from  the  point  where  an  exception  is  detected  to 
the  point  where  it  is  handled;  and  (d)  use  this  communication  path  to 
control  the  selection  of  relevant  exception  handlers.  It  is  assumed  that 
exceptions  occur  relatively  infrequently,  and  thus  the  cost  of  invoking  th<’ 
mechanism  for  signalling  and  handling  an  exception  can  be  high.  The 
co.'-t  of  both  entering  and  normally  exiting  a  body  of  code  that  contain.'' 
exc('ption  handlers  nuist  be  minimized. 

The  Lisp  languages  htivc  evolved  a  different  mechanism.  They  gemu 
ally  provide  a  means  of  naming  a  particular  control  state  (using  the  catch 
mechanism  in  CommoiiLisp,  for  example)  and  a  way  of  causing  an  arbi 
trary  value  to  be  returned  after  restoring  that  control  state  (throw).  These 
constructs  are  utilized  to  provide  many  forms  of  control  flow,  generally 
not  related  to  exceptional  conditions.  Typical  uses  include  iterators  and 
multiple-level  procedure  exit.  There  is  also  a  way  (unwind-protect)  to  as¬ 
sociate  with  a  block  of  code  a  set  of  actions  that  must  be  performed  when 
it  is  exited  in  either  manner  (the  normal  return  mechanism  or  the  throw 
method).  Unlike  the  other  group  of  languages,  Lisp  dialects  do  not  provide 
any  direct  method  of  disambigtiating  a  normal  return  from  an  exceptional 
case,  although  it  is  easy  enough  to  use  the  underlying  structure  to  devise  a 
mechanism  for  this  purpose. 

The  way  an  exception  condition  is  typically  handled  in  Lisp  is  to  provide 
a  ”w('ll-kn(nvn  name"  for  a  procedure  intended  to  handle  the  exceptional 
condition.  The  system  supplies  a  standard  handler  for  exceptions,  but 
users  are  free  to  rebind  the  name  (in  a  dynamically  scoped  Lisp)  or  provide 
an  alternative  fluid  value  (in  Scheme)  to  provide  customized  handling  of 
the  condition.  These  exception  handlers  are  called  in  such  a  way  that  the 
previous  value  for  the  exception  handler  is  always  available  under  the  stan¬ 
dard  name.  This  allows  an  exception  to  be  propagated  on  to  the  previous 
liandler.  Th('  control  How  can  be  altered  by  a  handler  in  the  same  way  it 
would  be  by  any  procedure,  using  throw  to  reestablish  an  existing  control 
"tate.  Normal  return  from  the  handler  may  either  attempt  to  continue  the 
com})utation  from  the  j)oint  where  the  exception  occurred  or  invoke  the 
previously  existing  handler,  depending  upon  the  j)articular  Lisp  system. 

The  two  mechanisms  appear  very  different  from  one  another,  and  yet 
there  is  a  deep  similarity.  There  are  a  number  of  features  that  can  be 
abstracted  from  both  of  these  methods  of  handling  exceptions  to  provide 


D.3.  CONTROL  STATE 


237 


a  basis  for  building  other  mechanisms.  The  essence  of  the  mechanisms 
appears  to  be  the  ability  to: 

1.  Mark  an  execution  state  for  possible  use  at  a  later  time. 

2.  Select  a  destination  execution  state  and  cause  a  transfer  of  control  to 
It. 

3.  Maintain  the  ordering  information  that  corresponds  to  the  normal 
return  path  between  marked  execution  states. 

4.  .■\s,s()ciate  transition  functions  with  marked  execution  states, 

d.  For  each  exception  condition  maintain  a  chain,  correlated  with  the 
return  path  chain,  of  (potential)  handlers  for  that  exception. 

G.  Pass  user  specified  information  about  the  exception  to  the  exception 
handh'rs. 

.MIT  Scheme  has  divided  these  problems  into  two  categories.  The  first 
involves  creating,  naming,  and  moving  among  ilifferent  jioints  in  the  exe¬ 
cution  of  a  program.  Collectively,  thesi*  operations  ( ntimliers  1  and  2)  refer 
to  what  is  known  as  the  control  state  of  the  program.  The  .second  is  related 
to  inserting,  removing,  locating,  and  executing  code  that  handles  exception 
conditions.  This  group  (numliers  4.  5.  and  C)  is  known  as  tin'  dynamic  state 
of  the  program.  The  two  are  not  indi'pi’iident .  and  it  is  in  the  intercon 
nection  of  the  two  (number  3)  that  the  iiK'chanisin  for  handling  exception 
|•onditions  lies. 


D.3  Control  State 

In  order  to  support  an  exci’ption  handling  mechanism  tla'ie  must  be  some 
way  of  marking,  at  run  time.  c<‘rtHin  points  in  the  execution  of  the  pro¬ 
gram  so  that  control  can  lie  restor'd  to  the.se  points  later.  Tlu'  notion, 
from  denotational  semantics,  of  a  continuation  r-mbodies  this  in  an  elegant 
mathematical  formulation.  The  continuation  <*ncapsulates  not  only  a  point 
in  the  code  (like  a  label  in  assemlily  language  or  BCPL).  l>ut  also  the  inher¬ 
ent  stack  of  operations  that  remains  to  be  iierformed  at  a  particular  jioint 
of  time  in  the  exc’cution  of  a  program.  Section  .\.4  describes  tin'  use  of 
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call-with-current-continuation  in  Scheme  to  capture  a  first-class  ()1>- 
jert  corresponding  the  the  current  continuation.  The  formal  semantics  of 
this  primitive  can  be  gleaned  from  the  denotational  description  of  Scheme 
contained  in  the  implementation-independent  language  definition [49]. 

This  mechanism  allows  any  point  in  the  execution  of  a  program  to  be 
labtded  (requirement  1  from  Section  D.2),  by  merely  making  the  continua¬ 
tion  in  existence  at  that  point  into  a  standard  Scheme  object.  The  selection 
of  an  ex('cution  state  to  be  resumed,  then,  is  identical  to  choosing  any  other 
object  in  the  system.  The  usual  naming  rules,  parameter  passing  rules,  and 
data  structures  can  be  used  to  devise  arbitrary  mechanisms.  Transfer  of 
control  to  the  cho.sen  state  is  accomplished  by  (at  least  syntactically)  pro¬ 
cedure  call,  since  continuation  objects  are  interchangeable  with  procedures 
(requirement  2). 

D.4  Dynamic  State 

MIT  Scheme  provides  a  mechanism  (proposed  by  Hemson  and  Lamping[28]) 
intended  to  support  requirements  4  and  5  of  Section  D.2.  A  dynamic  state 
space  (tlie  word  “dynamic”  is  often  omitted)  consists  of  a  number  of  state 
points  connected  by  arcs  with  a  pair  of  transition  functions  on  each  arc, 
one  for  each  direction  along  the  arc.  The  MIT  Scheme  system  is  always 
operating  at  a  specific  point  in  each  space  that  has  been  created.  Programs 
can  move  within  each  space,  causing  the  transition  functions  between  the 
original  j)osition  and  the  destination  point  to  be  executed.  In  order  to 
guarantee  a  unique  path  between  any  two  points  within  a  space,  the  space 
is  in  fact  constrained  to  be  a  tree. 

State  spaces  are  ordinary  Scheme  data  structures.  When  they  are  cre- 
aterl  initially  (by  make-state-space)  they  contain  a  single  point  in  which 
the  system  is  considered  to  be  operating.  They  can  be  extended  by  adding  a 
point  adjacent  to  the  current  one  (execute-at-new-state-point),  spec¬ 
ifying  the  transition  functions  into  and  out  of  that  point.  The  system 
executes  the  entering  transition  and  is  then  located  at  the  new  point.  Nor¬ 
mal  return  from  the  primitive  cavuses  a  transition  out  of  the  newly  created 
j)oint  and  into  the  point  where  execution  of  the  primitive  began.  And  a 
{irograrn  can  request  tramsfer  to  another  point  in  the  space  (translate- 
to-state-point ).  In  this  ca.se  a  path  is  calculated  to  the  destination  and 
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the  system  moves  a  step  at  a  time  toward  the  destination,  executing  a 
transition  function  at  each  step. 

The  state  space  abstraction  provides  a  convenient  packaging  of  the  no- 
titni  of  transition  functions,  providing  support  for  requirement  4.  Further¬ 
more.  the  state  space  is  built  in  a  manner  that  causes  it  to  reflect  the 
normal  return  path  for  programs  (unless  the  program  performs  a  translate- 
to-state-point ),  so  they  can  be  used  to  satisfy  requirement  5.  By  coupling 
the  use  of  translate-to-state-point  (motion  in  dynamic  state  space)  with 
the  use  of  continuations  (motion  in  control  space)  these  two  can  be  kept  in 
c(unj-)lete  agreement.  This  is  discussed  in  Section  D.5. 

The  current  dynamic  state  space  model  does  have  a  deficiency,  although 
it  is  easily  repaired.  There  is  currently  no  convenient  way  to  pass  infor¬ 
mation  along  the  chain  zrs  transitions  occur.  By  allowing  the  transition 
functions  to  receive  arguments  this  problem  would  be  solved,  allowing  re¬ 
quirement  6  to  be  easily  met. 


D.5  Connecting  the  States 

■At  the  lowest  level  of  the  MIT  Scheme  system  (in  the  “microcode”)  the 
control  space  and  the  dynamic  state  space  are  completely  independent  as 
described  so  far.  This  separation  hais  been  introduced  largely  to  allow 
exi)erinientation  with  mechanisms  that  support  the  exception  handling  at 
the  system  level.  For  ordinary  use,  however,  they  must  be  interconnected 
in  a  way  that  supports  the  common  use  of  the  system.  MIT  Scheme  uses 
;i  variant  on  the  familiar  Lisp  procedure  unwind-protect.  While  unwind- 
protect  alhnvs  a  region  of  code  to  have  an  exit  handler  attached  to  it,  this 
variant  (introduced  by  Stallman[52]  and  known  as  dyneunic-wind)  allows 
a  program  to  provide  transition  functions  that  will  be  invoked  upon  both 
(re-)entry  and  exit  of  a  block  of  code: 

(define  (dynamic-wind  re/entry  code  exit) 

(execute- at -new- St ate -point  sy stem- state- space 

re/entry 

code 

exit)) 


Dynamic-wind  operates  by  manipulating  a  specific  dynamic  state  space, 
system-state-space.  It  creates  a  new  point  within  that  space,  specifies 
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tlu'  function  re/entry  for  use  on  transition  into  that  point  and  exit  as  the 
transition  function  out  of  the  point.  It  moves  the  system  into  the  newly 
created  point  (causing  re/entry  to  be  evaluated)  and  then  executes  code. 
When  this  procedure  ends  the  system  will  return  to  the  previous  state 
point  (causing  exit  to  be  evaluated).  But,  by  itself,  this  will  not  cause 
control  transfers  that  occur  through  the  use  of  continuations  to  evaluate 
the  appropriate  thunks. 


In  order  to  make  dynamic-wind  a  standard  facility  the  continuation  ob¬ 
ject  that  users  normally  manipulate  is  not  just  the  control  state  as  might  be 
expected.  The  procedures  that  create  continuations  (such  as  call-with- 
current- continuation)  produce  procedures  (continuations)  that  capture 
both  the  control  state  and  the  current  point  within  the  system-state-space 
at  the  time  the  continuation  is  created.  When  this  procedure  is  called  (cor¬ 
responding  to  CommonLisp’s  throw  construct)  both  the  saved  control  state 
and  saved  dynamic  state  point  are  restored.  Since  restoring  the  dynamic 
state  point  is  done  by  moving  from  the  current  location  in  state  space  to 
the  one  being  restored  (using  translate-to-state-point)  all  of  the  appropriate 
transition  functions  are  invoked  in  the  process. 

The  introduction  of  parallel  processing  raises  a  problem  with  this  state 
space  model.  The  system  can  no  longer  be  considered  to  reside  in  a  single 
state  (control  or  dynamic)  at  each  instant  of  time.  The  notion  of  simul¬ 
taneously  being  in  multiple  states  must  be  considered.  The  control  state 
of  the  system  is  easily  captured  in  a  stack  and  allowing  each  processor  to 
provide  its  own  stack  conveniently  solves  this  problem.  The  dynamic  state, 
however,  is  characterized  by  arbitrary  user  specified  transition  functions. 
Since  transition  functions  always  come  in  pairs  (entry  and  exit),  a  single 
processor  can  move  (in  theory)  between  states  when  it  performs  a  task 
switch  and  return  to  the  same  set  of  conditions  it  was  in  when  it  left  the 
task.  But  the  states  may  be  mutually  exclusive  and  there  is  no  informa¬ 
tion  available  to  determine  which  states  are  compatible  and  which  are  not. 
By  separating  the  implementation  of  fluid-let  from  the  dynamic  space 
model  the  majority  of  problems  with  incompatible  states  are  avoided.  The 
remaining  mutually  incompatible  states  must  provide  transition  functions 
that  guard  against  any  simultaneous  access. 
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exception  after  the  block  has  been  entered. 

The  systein-state-space  is  always  correlated  with  the  execution  chain 
and  can  be  used  in  place  of  the  fluid  variables  to  implement  the  se¬ 
lection  of  an  exception  handler  and  propagation  of  the  exception.  In 
order  to  install  an  exception  handler,  dynamic-wind  is  u.sed  to  cre¬ 
ate  a  new  state  point.  A  pair  of  procedures  generated  in  a  standard 
way  are  installed  as  the  transition  functions.  These  procedures  have 
two  important  properties:  they  are  “easily  recognized”  as  exception 
handlers,  and  when  invoked  they  first  test  whether  an  exception  is 
being  signalled  and  then  whether  they  were  generated  to  handle  that 
exception.  Only  if  both  of  these  conditions  are  met  do  they  continue 
on  to  perform  their  work. 

When  an  exception  is  signalled,  the  system-state-space  is  walked  back 
from  the  current  point  of  execution  to  the  root  of  the  state  space. 
Each  transition  function  encountered  along  the  way  is  tested  to  see 
if  it  is  one  of  the  “ea.sily  recognized”  exception  handlers.  If  so,  it  is 
invoked  to  determine  whether  or  not  it  will  handle  the  exception.  If 
not  the  walk  continues  back  as  though  the  transition  had  not  been 
marked. 

This  approach  deals  well  with  the  addition  of  exception  conditions  on 
a  dynamic  basis.  Since  the  exception  handlers  are  invoked  for  every 
exception,  they  can  determine  at  the  time  when  the  exception  occurs 
whether  or  not  the  exception  is  one  to  which  they  apply.  Unfortu¬ 
nately.  it  requires  every  transition  to  be  examined,  even  ones  that  do 
not  apply  to  exception  handling. 

By  adding  a  new  state  space,  say  the  exception-state-space,  to  the  in¬ 
formation  that  is  saved  and  restored  with  a  continuation  it  is  possible 
to  allow  only  exception  handlers  to  be  examined  when  an  exception 
is  signalled.  The  mechanism  is  the  same  as  in  the  previous  approach, 
but  with  transitions  marked  in  this  alternate  space.  This  comes  very 
close  to  the  implementation  normally  found  in  languages  like  Mesa 
for  the  exception  mechanism. 

Tliis  imideinentation  does  have  an  efficiency  problem.  All  exception 
handlers  are  inspected,  even  ones  that  are  not  relevant  to  the  par¬ 
ticular  exception  being  initiated.  The  argument  generally  given  in 
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defense  of  this  is  that  exceptions  occur  infrequently  and  handlers  for 
exceptions  usually  include  a  default  hemdler,  so  the  inefficiency  is 
minimal. 

•  .\n  intriguing  alternative  to  this  last  approach  is  to  provide  a  sepa¬ 
rate  dynamic  state  space  for  each  kind  of  handler  that  is  installed.  A 
separate  data  structure  is  maintained  that  can  be  used  to  determine 
whether  a  particular  exception  needs  to  execute  any  transition  func¬ 
tions  in  that  space.  A  single  exception  condition  may  require  walking 
several  of  these  "exception  spaces,”  since  the  exception  may  satisfy 
more  than  one  criterion.  Thus  these  spaces  must  somehow  be  walked 
in  parallel  with  each  other. 

Installing  an  exception  handler  involves  choosing  the  correct  dynamic 
state  space  to  use.  Then  the  current  point  of  the  system  in  any  of 
the  other  exception  spaces  that  might  be  relevant  to  an  exception 
handled  by  this  space  must  be  captured.  The  transition  function 
installed  must  first  cause  the  point  in  each  of  these  other  spaces  to  be 
restored  to  the  captured  position  and  then  execute  the  actual  handler. 
This  intertwines  the  walks  of  the  trees  in  a  way  that  guarantees  that 
the  order  in  which  exception  handlers  are  examined  corresponds  to 
the  (reverse  of  the)  order  in  which  they  were  installed. 

By  calculating  which  state  spaces  are  relevant  when  an  exception  is 
signalled,  it  is  possible  to  examine  only  transitions  relevant  to  that 
exception.  Whether  this  elaborate  mechanism  is  more  efficient  than 
the  others  will  largely  depend  upon  the  extent  to  which  handlers  Eire 
created  and  removed  and  the  overlap  between  groups  of  ex 'eptions. 

Selecting  among  these  alternatives  is  a  matter  of  taste,  and  has  been  left 
to  users  of  the  MultiScheme  system.  As  mentioned  earlier,  no  current 
applications  (nor  the  system  itself)  have  required  an  exception  handling 
mechanism.  In  keeping  with  the  tradition  of  Lisp  languages,  when  need 
arises  for  such  a  mechctnism  the  system  implementors  and  the  users  of 
the  system  will  doubtless  negotiate  a  standard  interface  to  the  facilities 
described  here. 
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